| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-14 | |||
| 17:46:09 | dansmith | cfriesen: patches welcome | |
| 17:50:09 | cfriesen | dansmith: maybe next week. I'm now going to be busy modifying our automated upgrade tools and retesting everything. | |
| 17:56:01 | cfriesen | this won't impact people that upgrade nova-compute with all the guests still running....it's only a factor if you want to migrate all the instances off the "old" compute node to upgrade the OS as well. | |
| 17:59:51 | cfriesen | dansmith: question for you....if some online migrations can't complete until all the services are upgraded (like pcidevice_online_data_migration()) and others *must* complete in order to allow things to function, how does the person running the command know if it's safe to proceed? | |
| 18:01:15 | cfriesen | dansmith: it makes it really hard to reliably automate the call to "nova-manage db online_data_migrations" | |
| 18:02:02 | cfriesen | seems to me we need to break it up into two calls, one right after the control services are upgraded, and one after all the compute nodes are also upgraded | |
| 18:03:17 | dansmith | we've only ever had that one thing that needed to run after computes are up, and it's gone now | |
| 18:03:41 | dansmith | it's a very exceptional case and once you did it on any release it was present on, it was done | |
| 18:03:53 | dansmith | so not in favor of complicating things further based on one exceptional example | |
| 18:04:05 | dansmith | it could have printed a warning explanation and been fine I think | |
| 18:04:21 | dansmith | "sorry homey, I can't finish until after, so blah blah" | |
| 18:04:22 | cfriesen | dansmith: okay, didn't realize that was the only thing ever | |
| 18:04:52 | dansmith | yep, the only one ever, which is why in my head, you do this when control is upgraded | |
| 18:21:29 | mnaser | i'm trying to do some perf tweaking and i'm just curious on why a relatively idle 60-ish node cloud conductor rabbitmq message rates is at ~20/s | |
| 18:21:52 | mnaser | i increased the service report interval from 10 seconds up to 60s | |
| 18:22:13 | mnaser | and why i say relatively idle, there is no new servers being added/deleted etc | |
| 18:27:15 | mriedem | melwitt: one small thing in here https://review.openstack.org/#/c/446241/ | |
| 18:27:26 | mriedem | melwitt: in a normal network delete, we call network_api.disassociate and then delete, | |
| 18:27:32 | mriedem | so we should do the same for that rollback | |
| 18:27:43 | mriedem | melwitt: i also found a fun latent networks quota bug in there | |
| 18:28:56 | mriedem | mnaser: could be a few things, periodic tasks in the computes come to mine | |
| 18:28:58 | mriedem | *mind | |
| 18:29:02 | melwitt | mriedem: cool, thanks. looking | |
| 18:29:14 | mriedem | might also depend on if you have instance_usage_audit enabled to send notifications | |
| 18:29:39 | edleafe | mnaser: You should probably ask on the #openstack channel. See #topic | |
| 18:30:02 | mriedem | edleafe: i actually want to know what mnaser is doing and finding out, given he runs vexxhost :) | |
| 18:30:09 | mnaser | edleafe im aware, i was just looking more on the internals of nova, i wasn't looking for a "how to fix" but more of a "what components would be good to look at" | |
| 18:30:48 | mriedem | mnaser: the computes are going to be running periodic tasks and pulling instances from the database over conductor | |
| 18:30:49 | mnaser | mriedem we toned down the service check ins (which is used by the services / scheduler) to 1 minute + 4 minute service_down_time and that helped | |
| 18:30:51 | mriedem | which is going to be rpc traffic | |
| 18:31:00 | mriedem | mnaser: this isn't the service check in | |
| 18:31:01 | mnaser | but yeah, periodic tasks is not something i've touched on yet | |
| 18:31:03 | mriedem | with the service group api | |
| 18:31:14 | mriedem | this would be things like the update_available_resources periodic task in the compute | |
| 18:31:36 | mriedem | that runs every minute by default to update resource usage information in the compute_nodes table and in the placement api | |
| 18:31:40 | mriedem | mnaser: which release? | |
| 18:31:53 | mriedem | there is also the instance_usage_audit task but that's disabled by default | |
| 18:31:59 | mnaser | i'll throw up debugging on a compute node | |
| 18:32:00 | mnaser | newton atm, no placement service in place yet | |
| 18:32:05 | mriedem | ok | |
| 18:32:10 | mnaser | yeah instance_usage_audit is off (well, i havent verified but notifications queue is empty and no consumers for it) | |
| 18:32:15 | mriedem | even w/o placement there is probably a lot of steady state traffic | |
| 18:32:35 | mriedem | the computes are also sending instance info to the filter scheduler every minute | |
| 18:32:37 | mriedem | *all* computes | |
| 18:32:42 | mriedem | even if you're not using the filter scheduler | |
| 18:32:45 | mriedem | you're welcome | |
| 18:32:53 | mnaser | lol! | |
| 18:33:12 | mriedem | are you using the filter scheduler or caching scheduler? | |
| 18:33:39 | mnaser | we are so i guess that helps, but 60 nodes every minute means 60 updates only so that averages to just one per second (and i think fuzzing in place in nova can help alleviate the big burst) | |
| 18:33:43 | mnaser | caching scheduler...? til. | |
| 18:33:56 | mnaser | the filter does the job for us though we had to reorder things because it took too long sometimes | |
| 18:34:13 | mnaser | aka computefilter is the last one but by the time it runs, "service is down" | |
| 18:34:50 | mnaser | (i dont know why it isn't earlier in by default, considering it's one of the more fundamental ones that might drop a whole bunch) | |
| 18:35:04 | mriedem | these are the current defaults | |
| 18:35:06 | mriedem | default=[ | |
| 18:35:06 | mriedem | "RetryFilter", | |
| 18:35:06 | mriedem | "AvailabilityZoneFilter", | |
| 18:35:06 | mriedem | "RamFilter", | |
| 18:35:06 | mriedem | "DiskFilter", | |
| 18:35:06 | mriedem | "ComputeFilter", | |
| 18:35:06 | mriedem | "ComputeCapabilitiesFilter", | |
| 18:35:07 | mriedem | "ImagePropertiesFilter", | |
| 18:35:07 | mriedem | "ServerGroupAntiAffinityFilter", | |
| 18:35:08 | mriedem | "ServerGroupAffinityFilter", | |
| 18:35:08 | mriedem | ], | |
| 18:35:36 | mriedem | i think we can drop ramfilter and diskfilter from defaults now that we have placement required in ocata, but would have to check that out again | |
| 18:35:59 | mnaser | computefilter pretty much checks if service is alive | |
| 18:36:00 | mnaser | so... id imagine that's more important than figuring out if you have enough ram | |
| 18:36:22 | mnaser | but i guess the logic was ram will eliminate a bigger subset of possiblities rather than disabled nodes | |
| 18:36:31 | mnaser | but yeah, placement mix address a whole bunch of that | |
| 18:36:33 | mriedem | assuming there was logic | |
| 18:36:54 | mriedem | mnaser: anyway, let us know if you find anything | |
| 18:37:03 | mriedem | i always like to hear about performance issues | |
| 18:37:33 | mnaser | we've had some neutron ones but so far with nova, it's just trying to see how we can minimize rpc traffic so to handle big loads | |
| 18:37:51 | mnaser | we had the pleasure of an api request with num_instances => 200 | |
| 18:38:31 | mnaser | nova seemed to cope well, but neutron couldn't catch up (more specifically neutron-dhcp-agent), so at some point instance port plugging started timing out (vif_plugging_timeout was getting hit) | |
| 18:40:28 | mnaser | which did raise the issue of.. there is no way of limiting num_instances in api requests (based on my research of the code base), so if the quota is 100 vms.. with one api request you can cause a lot of trouble (rate limiting wouldn't be a solution here considering its a single request) | |
| 18:41:23 | mriedem | eesh that's a good point | |
| 18:41:50 | mriedem | i thought there was a port quota in neutron of 50 by default? | |
| 18:42:05 | mnaser | right, but we naturally raise the port quota when we increase the VM quota anyways | |
| 18:42:08 | mriedem | the compute api checks with neutron how many ports can be created in a single multi-instance create run | |
| 18:42:18 | mriedem | ok | |
| 18:42:45 | mnaser | so say you have 200 vms quota, 200 ports quota in neutron, you can send a request with num_instances = 200 and things can get a bit messy | |
| 18:42:52 | mriedem | i'd have to look but i think if you ask for 200 but you can only get 100, we drop the count down to 100 based on how many ports neutron will let you create | |
| 18:43:11 | mriedem | but then it's a race to the computes which actually create the ports | |
| 18:43:29 | mnaser | thing is: i'm okay with a customer having 200 vms, it's just the "create them all at once" thing that ends up hurting a bit | |
| 18:44:22 | mnaser | because then not only do their vms end up in error states but all new vms timeout in spawning until neutron-dhcp-agent catches up (because reload_allocations does 1 network at a time, one port update at a time) | |
| 18:44:23 | mriedem | yeah we don't have any cap on how many instances you can ask for in the REST API schema itself | |
| 18:45:10 | mriedem | melwitt: speaking of api quota things, and your networks quota patch, do you remember if you can even POST /os-tenant-networks in cells v1? | |
| 18:45:12 | mnaser | so the dhcp agent queue grows (and the problem gets even worse when these vms fail, all the ports that failed to get activated receive a delete request, and you just doubled the size of that queue as the dhcp agent is updated about "deleting" them) | |
| 18:45:26 | mnaser | but that's more of a neutron side of things i'm hoping to bring up | |
| 18:45:56 | mriedem | that will also cause api traffice between nova and neutron when the ports are deleted | |
| 18:46:00 | melwitt | mriedem: I think you can, it just won't propagate to child cells at all, so would not result in any instance being able to use the new network if you added one | |
| 18:46:23 | mriedem | nova deletes the port, neutron sends an api request back to nova saying something change via the os-server-external-events api, which rpc casts to the compute to do some cleanup | |
| 18:47:35 | mriedem | melwitt: so you can, but it doesn't work | |
| 18:47:35 | mnaser | yeah, it just causes a lot of bad times overall | |
| 18:47:37 | mriedem | so you can't :) | |
| 18:47:59 | mnaser | workaround would be to bring vif_plugging_timeout up but .. 300 seconds is already long enough :x | |
| 18:48:02 | mriedem | melwitt: i see we exclude tempest test test_list_show_tenant_networks from the cells v1 job | |
| 18:49:17 | mriedem | melwitt: i just got thinking about that because of my rubiks cube questions in that change about assuming single cell for getting all networks | |