| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-14 | |||
| 17:35:54 | dansmith | to insulate computes from those changes | |
| 17:40:40 | cfriesen | dansmith: okay. that answers my question, but I think we need to update the docs at https://docs.openstack.org/developer/nova/upgrade.html to make it clear that the online data migrations should be run earlier *during* the maintenance window. | |
| 17:41:09 | cfriesen | dansmith: and remove the comments that talk about it only needing to be run before the upgrade to the *next* version | |
| 17:41:43 | dansmith | cfriesen: perhaps mentioning that if you fail to reduce the remaining updates past a certain threshold, those may be things that need to run again later | |
| 17:42:52 | openstackgerrit | melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634 | |
| 17:42:53 | openstackgerrit | melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635 | |
| 17:42:58 | dansmith | I probably wrote that text and was writing it attempting to hit a generic scenario without mentioning every possibility, where I just put it in the process somewhere | |
| 17:42:58 | cfriesen | dansmith: doesn't anything greater than zero mean you need to re-run it later? | |
| 17:43:14 | dansmith | cfriesen: if you run with --until-done or whatever, yeah | |
| 17:43:44 | cfriesen | dansmith: originally I think the idea was that you'd only need to run the online_data_migrations sometime before the next upgrade...that's the way it's written at https://docs.openstack.org/releasenotes/nova/newton.html | |
| 17:43:57 | cfriesen | "... budget time to run these online migrations before you plan to upgrade to ocata" | |
| 17:44:02 | dansmith | cfriesen: that has never been the "design intent" | |
| 17:44:32 | dansmith | because as those docs say, you're expecting more conductor load during the period where you're not fully migrated, so budget resources for that | |
| 17:45:29 | cfriesen | dansmith: there's been a disconnect then, because that sort of wording is all over the upstream documentation around upgrades | |
| 17:46:09 | dansmith | cfriesen: patches welcome | |
| 17:50:09 | cfriesen | dansmith: maybe next week. I'm now going to be busy modifying our automated upgrade tools and retesting everything. | |
| 17:56:01 | cfriesen | this won't impact people that upgrade nova-compute with all the guests still running....it's only a factor if you want to migrate all the instances off the "old" compute node to upgrade the OS as well. | |
| 17:59:51 | cfriesen | dansmith: question for you....if some online migrations can't complete until all the services are upgraded (like pcidevice_online_data_migration()) and others *must* complete in order to allow things to function, how does the person running the command know if it's safe to proceed? | |
| 18:01:15 | cfriesen | dansmith: it makes it really hard to reliably automate the call to "nova-manage db online_data_migrations" | |
| 18:02:02 | cfriesen | seems to me we need to break it up into two calls, one right after the control services are upgraded, and one after all the compute nodes are also upgraded | |
| 18:03:17 | dansmith | we've only ever had that one thing that needed to run after computes are up, and it's gone now | |
| 18:03:41 | dansmith | it's a very exceptional case and once you did it on any release it was present on, it was done | |
| 18:03:53 | dansmith | so not in favor of complicating things further based on one exceptional example | |
| 18:04:05 | dansmith | it could have printed a warning explanation and been fine I think | |
| 18:04:21 | dansmith | "sorry homey, I can't finish until after, so blah blah" | |
| 18:04:22 | cfriesen | dansmith: okay, didn't realize that was the only thing ever | |
| 18:04:52 | dansmith | yep, the only one ever, which is why in my head, you do this when control is upgraded | |
| 18:21:29 | mnaser | i'm trying to do some perf tweaking and i'm just curious on why a relatively idle 60-ish node cloud conductor rabbitmq message rates is at ~20/s | |
| 18:21:52 | mnaser | i increased the service report interval from 10 seconds up to 60s | |
| 18:22:13 | mnaser | and why i say relatively idle, there is no new servers being added/deleted etc | |
| 18:27:15 | mriedem | melwitt: one small thing in here https://review.openstack.org/#/c/446241/ | |
| 18:27:26 | mriedem | melwitt: in a normal network delete, we call network_api.disassociate and then delete, | |
| 18:27:32 | mriedem | so we should do the same for that rollback | |
| 18:27:43 | mriedem | melwitt: i also found a fun latent networks quota bug in there | |
| 18:28:56 | mriedem | mnaser: could be a few things, periodic tasks in the computes come to mine | |
| 18:28:58 | mriedem | *mind | |
| 18:29:02 | melwitt | mriedem: cool, thanks. looking | |
| 18:29:14 | mriedem | might also depend on if you have instance_usage_audit enabled to send notifications | |
| 18:29:39 | edleafe | mnaser: You should probably ask on the #openstack channel. See #topic | |
| 18:30:02 | mriedem | edleafe: i actually want to know what mnaser is doing and finding out, given he runs vexxhost :) | |
| 18:30:09 | mnaser | edleafe im aware, i was just looking more on the internals of nova, i wasn't looking for a "how to fix" but more of a "what components would be good to look at" | |
| 18:30:48 | mriedem | mnaser: the computes are going to be running periodic tasks and pulling instances from the database over conductor | |
| 18:30:49 | mnaser | mriedem we toned down the service check ins (which is used by the services / scheduler) to 1 minute + 4 minute service_down_time and that helped | |
| 18:30:51 | mriedem | which is going to be rpc traffic | |
| 18:31:00 | mriedem | mnaser: this isn't the service check in | |
| 18:31:01 | mnaser | but yeah, periodic tasks is not something i've touched on yet | |
| 18:31:03 | mriedem | with the service group api | |
| 18:31:14 | mriedem | this would be things like the update_available_resources periodic task in the compute | |
| 18:31:36 | mriedem | that runs every minute by default to update resource usage information in the compute_nodes table and in the placement api | |
| 18:31:40 | mriedem | mnaser: which release? | |
| 18:31:53 | mriedem | there is also the instance_usage_audit task but that's disabled by default | |
| 18:31:59 | mnaser | i'll throw up debugging on a compute node | |
| 18:32:00 | mnaser | newton atm, no placement service in place yet | |
| 18:32:05 | mriedem | ok | |
| 18:32:10 | mnaser | yeah instance_usage_audit is off (well, i havent verified but notifications queue is empty and no consumers for it) | |
| 18:32:15 | mriedem | even w/o placement there is probably a lot of steady state traffic | |
| 18:32:35 | mriedem | the computes are also sending instance info to the filter scheduler every minute | |
| 18:32:37 | mriedem | *all* computes | |
| 18:32:42 | mriedem | even if you're not using the filter scheduler | |
| 18:32:45 | mriedem | you're welcome | |
| 18:32:53 | mnaser | lol! | |
| 18:33:12 | mriedem | are you using the filter scheduler or caching scheduler? | |
| 18:33:39 | mnaser | we are so i guess that helps, but 60 nodes every minute means 60 updates only so that averages to just one per second (and i think fuzzing in place in nova can help alleviate the big burst) | |
| 18:33:43 | mnaser | caching scheduler...? til. | |
| 18:33:56 | mnaser | the filter does the job for us though we had to reorder things because it took too long sometimes | |
| 18:34:13 | mnaser | aka computefilter is the last one but by the time it runs, "service is down" | |
| 18:34:50 | mnaser | (i dont know why it isn't earlier in by default, considering it's one of the more fundamental ones that might drop a whole bunch) | |
| 18:35:04 | mriedem | these are the current defaults | |
| 18:35:06 | mriedem | "ComputeCapabilitiesFilter", | |
| 18:35:06 | mriedem | "ComputeFilter", | |
| 18:35:06 | mriedem | "DiskFilter", | |
| 18:35:06 | mriedem | "RamFilter", | |
| 18:35:06 | mriedem | "AvailabilityZoneFilter", | |
| 18:35:06 | mriedem | "RetryFilter", | |
| 18:35:06 | mriedem | default=[ | |
| 18:35:07 | mriedem | "ServerGroupAntiAffinityFilter", | |
| 18:35:07 | mriedem | "ImagePropertiesFilter", | |
| 18:35:08 | mriedem | ], | |
| 18:35:08 | mriedem | "ServerGroupAffinityFilter", | |
| 18:35:36 | mriedem | i think we can drop ramfilter and diskfilter from defaults now that we have placement required in ocata, but would have to check that out again | |
| 18:35:59 | mnaser | computefilter pretty much checks if service is alive | |
| 18:36:00 | mnaser | so... id imagine that's more important than figuring out if you have enough ram | |
| 18:36:22 | mnaser | but i guess the logic was ram will eliminate a bigger subset of possiblities rather than disabled nodes | |
| 18:36:31 | mnaser | but yeah, placement mix address a whole bunch of that | |
| 18:36:33 | mriedem | assuming there was logic | |
| 18:36:54 | mriedem | mnaser: anyway, let us know if you find anything | |
| 18:37:03 | mriedem | i always like to hear about performance issues | |
| 18:37:33 | mnaser | we've had some neutron ones but so far with nova, it's just trying to see how we can minimize rpc traffic so to handle big loads | |
| 18:37:51 | mnaser | we had the pleasure of an api request with num_instances => 200 | |
| 18:38:31 | mnaser | nova seemed to cope well, but neutron couldn't catch up (more specifically neutron-dhcp-agent), so at some point instance port plugging started timing out (vif_plugging_timeout was getting hit) | |
| 18:40:28 | mnaser | which did raise the issue of.. there is no way of limiting num_instances in api requests (based on my research of the code base), so if the quota is 100 vms.. with one api request you can cause a lot of trouble (rate limiting wouldn't be a solution here considering its a single request) | |
| 18:41:23 | mriedem | eesh that's a good point | |
| 18:41:50 | mriedem | i thought there was a port quota in neutron of 50 by default? | |
| 18:42:05 | mnaser | right, but we naturally raise the port quota when we increase the VM quota anyways | |
| 18:42:08 | mriedem | the compute api checks with neutron how many ports can be created in a single multi-instance create run | |
| 18:42:18 | mriedem | ok | |
| 18:42:45 | mnaser | so say you have 200 vms quota, 200 ports quota in neutron, you can send a request with num_instances = 200 and things can get a bit messy | |
| 18:42:52 | mriedem | i'd have to look but i think if you ask for 200 but you can only get 100, we drop the count down to 100 based on how many ports neutron will let you create | |
| 18:43:11 | mriedem | but then it's a race to the computes which actually create the ports | |
| 18:43:29 | mnaser | thing is: i'm okay with a customer having 200 vms, it's just the "create them all at once" thing that ends up hurting a bit | |