| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-14 | |||
| 18:21:52 | mnaser | i increased the service report interval from 10 seconds up to 60s | |
| 18:22:13 | mnaser | and why i say relatively idle, there is no new servers being added/deleted etc | |
| 18:27:15 | mriedem | melwitt: one small thing in here https://review.openstack.org/#/c/446241/ | |
| 18:27:26 | mriedem | melwitt: in a normal network delete, we call network_api.disassociate and then delete, | |
| 18:27:32 | mriedem | so we should do the same for that rollback | |
| 18:27:43 | mriedem | melwitt: i also found a fun latent networks quota bug in there | |
| 18:28:56 | mriedem | mnaser: could be a few things, periodic tasks in the computes come to mine | |
| 18:28:58 | mriedem | *mind | |
| 18:29:02 | melwitt | mriedem: cool, thanks. looking | |
| 18:29:14 | mriedem | might also depend on if you have instance_usage_audit enabled to send notifications | |
| 18:29:39 | edleafe | mnaser: You should probably ask on the #openstack channel. See #topic | |
| 18:30:02 | mriedem | edleafe: i actually want to know what mnaser is doing and finding out, given he runs vexxhost :) | |
| 18:30:09 | mnaser | edleafe im aware, i was just looking more on the internals of nova, i wasn't looking for a "how to fix" but more of a "what components would be good to look at" | |
| 18:30:48 | mriedem | mnaser: the computes are going to be running periodic tasks and pulling instances from the database over conductor | |
| 18:30:49 | mnaser | mriedem we toned down the service check ins (which is used by the services / scheduler) to 1 minute + 4 minute service_down_time and that helped | |
| 18:30:51 | mriedem | which is going to be rpc traffic | |
| 18:31:00 | mriedem | mnaser: this isn't the service check in | |
| 18:31:01 | mnaser | but yeah, periodic tasks is not something i've touched on yet | |
| 18:31:03 | mriedem | with the service group api | |
| 18:31:14 | mriedem | this would be things like the update_available_resources periodic task in the compute | |
| 18:31:36 | mriedem | that runs every minute by default to update resource usage information in the compute_nodes table and in the placement api | |
| 18:31:40 | mriedem | mnaser: which release? | |
| 18:31:53 | mriedem | there is also the instance_usage_audit task but that's disabled by default | |
| 18:31:59 | mnaser | i'll throw up debugging on a compute node | |
| 18:32:00 | mnaser | newton atm, no placement service in place yet | |
| 18:32:05 | mriedem | ok | |
| 18:32:10 | mnaser | yeah instance_usage_audit is off (well, i havent verified but notifications queue is empty and no consumers for it) | |
| 18:32:15 | mriedem | even w/o placement there is probably a lot of steady state traffic | |
| 18:32:35 | mriedem | the computes are also sending instance info to the filter scheduler every minute | |
| 18:32:37 | mriedem | *all* computes | |
| 18:32:42 | mriedem | even if you're not using the filter scheduler | |
| 18:32:45 | mriedem | you're welcome | |
| 18:32:53 | mnaser | lol! | |
| 18:33:12 | mriedem | are you using the filter scheduler or caching scheduler? | |
| 18:33:39 | mnaser | we are so i guess that helps, but 60 nodes every minute means 60 updates only so that averages to just one per second (and i think fuzzing in place in nova can help alleviate the big burst) | |
| 18:33:43 | mnaser | caching scheduler...? til. | |
| 18:33:56 | mnaser | the filter does the job for us though we had to reorder things because it took too long sometimes | |
| 18:34:13 | mnaser | aka computefilter is the last one but by the time it runs, "service is down" | |
| 18:34:50 | mnaser | (i dont know why it isn't earlier in by default, considering it's one of the more fundamental ones that might drop a whole bunch) | |
| 18:35:04 | mriedem | these are the current defaults | |
| 18:35:06 | mriedem | "ComputeCapabilitiesFilter", | |
| 18:35:06 | mriedem | "ComputeFilter", | |
| 18:35:06 | mriedem | "DiskFilter", | |
| 18:35:06 | mriedem | "RamFilter", | |
| 18:35:06 | mriedem | "AvailabilityZoneFilter", | |
| 18:35:06 | mriedem | "RetryFilter", | |
| 18:35:06 | mriedem | default=[ | |
| 18:35:07 | mriedem | "ServerGroupAntiAffinityFilter", | |
| 18:35:07 | mriedem | "ImagePropertiesFilter", | |
| 18:35:08 | mriedem | ], | |
| 18:35:08 | mriedem | "ServerGroupAffinityFilter", | |
| 18:35:36 | mriedem | i think we can drop ramfilter and diskfilter from defaults now that we have placement required in ocata, but would have to check that out again | |
| 18:35:59 | mnaser | computefilter pretty much checks if service is alive | |
| 18:36:00 | mnaser | so... id imagine that's more important than figuring out if you have enough ram | |
| 18:36:22 | mnaser | but i guess the logic was ram will eliminate a bigger subset of possiblities rather than disabled nodes | |
| 18:36:31 | mnaser | but yeah, placement mix address a whole bunch of that | |
| 18:36:33 | mriedem | assuming there was logic | |
| 18:36:54 | mriedem | mnaser: anyway, let us know if you find anything | |
| 18:37:03 | mriedem | i always like to hear about performance issues | |
| 18:37:33 | mnaser | we've had some neutron ones but so far with nova, it's just trying to see how we can minimize rpc traffic so to handle big loads | |
| 18:37:51 | mnaser | we had the pleasure of an api request with num_instances => 200 | |
| 18:38:31 | mnaser | nova seemed to cope well, but neutron couldn't catch up (more specifically neutron-dhcp-agent), so at some point instance port plugging started timing out (vif_plugging_timeout was getting hit) | |
| 18:40:28 | mnaser | which did raise the issue of.. there is no way of limiting num_instances in api requests (based on my research of the code base), so if the quota is 100 vms.. with one api request you can cause a lot of trouble (rate limiting wouldn't be a solution here considering its a single request) | |
| 18:41:23 | mriedem | eesh that's a good point | |
| 18:41:50 | mriedem | i thought there was a port quota in neutron of 50 by default? | |
| 18:42:05 | mnaser | right, but we naturally raise the port quota when we increase the VM quota anyways | |
| 18:42:08 | mriedem | the compute api checks with neutron how many ports can be created in a single multi-instance create run | |
| 18:42:18 | mriedem | ok | |
| 18:42:45 | mnaser | so say you have 200 vms quota, 200 ports quota in neutron, you can send a request with num_instances = 200 and things can get a bit messy | |
| 18:42:52 | mriedem | i'd have to look but i think if you ask for 200 but you can only get 100, we drop the count down to 100 based on how many ports neutron will let you create | |
| 18:43:11 | mriedem | but then it's a race to the computes which actually create the ports | |
| 18:43:29 | mnaser | thing is: i'm okay with a customer having 200 vms, it's just the "create them all at once" thing that ends up hurting a bit | |
| 18:44:22 | mnaser | because then not only do their vms end up in error states but all new vms timeout in spawning until neutron-dhcp-agent catches up (because reload_allocations does 1 network at a time, one port update at a time) | |
| 18:44:23 | mriedem | yeah we don't have any cap on how many instances you can ask for in the REST API schema itself | |
| 18:45:10 | mriedem | melwitt: speaking of api quota things, and your networks quota patch, do you remember if you can even POST /os-tenant-networks in cells v1? | |
| 18:45:12 | mnaser | so the dhcp agent queue grows (and the problem gets even worse when these vms fail, all the ports that failed to get activated receive a delete request, and you just doubled the size of that queue as the dhcp agent is updated about "deleting" them) | |
| 18:45:26 | mnaser | but that's more of a neutron side of things i'm hoping to bring up | |
| 18:45:56 | mriedem | that will also cause api traffice between nova and neutron when the ports are deleted | |
| 18:46:00 | melwitt | mriedem: I think you can, it just won't propagate to child cells at all, so would not result in any instance being able to use the new network if you added one | |
| 18:46:23 | mriedem | nova deletes the port, neutron sends an api request back to nova saying something change via the os-server-external-events api, which rpc casts to the compute to do some cleanup | |
| 18:47:35 | mnaser | yeah, it just causes a lot of bad times overall | |
| 18:47:35 | mriedem | melwitt: so you can, but it doesn't work | |
| 18:47:37 | mriedem | so you can't :) | |
| 18:47:59 | mnaser | workaround would be to bring vif_plugging_timeout up but .. 300 seconds is already long enough :x | |
| 18:48:02 | mriedem | melwitt: i see we exclude tempest test test_list_show_tenant_networks from the cells v1 job | |
| 18:49:17 | mriedem | melwitt: i just got thinking about that because of my rubiks cube questions in that change about assuming single cell for getting all networks | |
| 18:49:33 | mriedem | took me to a dark place about cells v1 and cells v2 and nova-network and what won't work | |
| 18:50:08 | melwitt | mriedem: that stuff is nova-network only too, I think. the create/delete network stuff | |
| 18:51:45 | mriedem | melwitt: yes for os-tenant-networks it is | |
| 18:52:33 | melwitt | yeah | |
| 18:56:28 | mriedem | dansmith: whenever we declare support for multiple v2 cells in the release notes, we should remember to mention that excludes nova-network https://review.openstack.org/#/c/446241/20/nova/api/openstack/compute/tenant_networks.py@200 | |
| 19:34:22 | dansmith | mriedem: totes | |
| 19:59:39 | cfriesen | dansmith: this may be an issue for bauzas, but I had an entry in the DB for the "pci_requests" column in the "instance_extra" table. I then ran the online DB migrations which generated a new RequestSpec for the instance, but the generated RequestSpec for this instance has an emtpy list of pci requests and the "pci_requests" column in "instance_extra" is now empty | |
| 20:03:49 | melwitt | mriedem: replied on the networks quota patch. it seems to me that disassociate is only needed if associate could ever have happened and in the recheck_quota case, it couldn't have. do you think we should still do disassociate anyway? | |
| 20:07:51 | mriedem | melwitt: is that the POST /os-networks/{network_id}/action "associate_host" action? | |
| 20:09:41 | melwitt | lemme see ... whatever calls the nova-network manager's associate method | |
| 20:09:47 | mriedem | os-networks-associate | |
| 20:10:07 | melwitt | this is so confusing | |
| 20:10:58 | melwitt | anyway, I'm not opposed to adding the disassociate call, I just noticed I don't think it would ever do anything | |
| 20:11:06 | mriedem | no i think you're right | |