Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-14
17:33:01 dansmith after which you can roll your computes later
17:33:53 dansmith although it's also confusing the situation with the config bits being before that
17:33:54 cfriesen dansmith: it can't be the maintenance window for just control services, it talks about bumping the RPC version
17:34:10 cfriesen "Once all the services are running the latest version of the code, and all the services are aware they all have been upgraded, it is safe to transform the data in the database into its new format."
17:34:12 dansmith yeah, its' confusing
17:34:57 dansmith normally you wouldn't really have to run it twice, and either after control or after everything is done would be enough,
17:35:14 dansmith but if you've got a release where there are critical things on both sides...
17:35:38 cfriesen dansmith: do we guarantee that the online data migrations won't change things in such a way that they break old compute nodes?
17:35:50 dansmith that's the whole point of conductor
17:35:54 dansmith to insulate computes from those changes
17:40:40 cfriesen dansmith: okay. that answers my question, but I think we need to update the docs at https://docs.openstack.org/developer/nova/upgrade.html to make it clear that the online data migrations should be run earlier *during* the maintenance window.
17:41:09 cfriesen dansmith: and remove the comments that talk about it only needing to be run before the upgrade to the *next* version
17:41:43 dansmith cfriesen: perhaps mentioning that if you fail to reduce the remaining updates past a certain threshold, those may be things that need to run again later
17:42:52 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
17:42:53 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
17:42:58 dansmith I probably wrote that text and was writing it attempting to hit a generic scenario without mentioning every possibility, where I just put it in the process somewhere
17:42:58 cfriesen dansmith: doesn't anything greater than zero mean you need to re-run it later?
17:43:14 dansmith cfriesen: if you run with --until-done or whatever, yeah
17:43:44 cfriesen dansmith: originally I think the idea was that you'd only need to run the online_data_migrations sometime before the next upgrade...that's the way it's written at https://docs.openstack.org/releasenotes/nova/newton.html
17:43:57 cfriesen "... budget time to run these online migrations before you plan to upgrade to ocata"
17:44:02 dansmith cfriesen: that has never been the "design intent"
17:44:32 dansmith because as those docs say, you're expecting more conductor load during the period where you're not fully migrated, so budget resources for that
17:45:29 cfriesen dansmith: there's been a disconnect then, because that sort of wording is all over the upstream documentation around upgrades
17:46:09 dansmith cfriesen: patches welcome
17:50:09 cfriesen dansmith: maybe next week. I'm now going to be busy modifying our automated upgrade tools and retesting everything.
17:56:01 cfriesen this won't impact people that upgrade nova-compute with all the guests still running....it's only a factor if you want to migrate all the instances off the "old" compute node to upgrade the OS as well.
17:59:51 cfriesen dansmith: question for you....if some online migrations can't complete until all the services are upgraded (like pcidevice_online_data_migration()) and others *must* complete in order to allow things to function, how does the person running the command know if it's safe to proceed?
18:01:15 cfriesen dansmith: it makes it really hard to reliably automate the call to "nova-manage db online_data_migrations"
18:02:02 cfriesen seems to me we need to break it up into two calls, one right after the control services are upgraded, and one after all the compute nodes are also upgraded
18:03:17 dansmith we've only ever had that one thing that needed to run after computes are up, and it's gone now
18:03:41 dansmith it's a very exceptional case and once you did it on any release it was present on, it was done
18:03:53 dansmith so not in favor of complicating things further based on one exceptional example
18:04:05 dansmith it could have printed a warning explanation and been fine I think
18:04:21 dansmith "sorry homey, I can't finish until after, so blah blah"
18:04:22 cfriesen dansmith: okay, didn't realize that was the only thing ever
18:04:52 dansmith yep, the only one ever, which is why in my head, you do this when control is upgraded
18:21:29 mnaser i'm trying to do some perf tweaking and i'm just curious on why a relatively idle 60-ish node cloud conductor rabbitmq message rates is at ~20/s
18:21:52 mnaser i increased the service report interval from 10 seconds up to 60s
18:22:13 mnaser and why i say relatively idle, there is no new servers being added/deleted etc
18:27:15 mriedem melwitt: one small thing in here https://review.openstack.org/#/c/446241/
18:27:26 mriedem melwitt: in a normal network delete, we call network_api.disassociate and then delete,
18:27:32 mriedem so we should do the same for that rollback
18:27:43 mriedem melwitt: i also found a fun latent networks quota bug in there
18:28:56 mriedem mnaser: could be a few things, periodic tasks in the computes come to mine
18:28:58 mriedem *mind
18:29:02 melwitt mriedem: cool, thanks. looking
18:29:14 mriedem might also depend on if you have instance_usage_audit enabled to send notifications
18:29:39 edleafe mnaser: You should probably ask on the #openstack channel. See #topic
18:30:02 mriedem edleafe: i actually want to know what mnaser is doing and finding out, given he runs vexxhost :)
18:30:09 mnaser edleafe im aware, i was just looking more on the internals of nova, i wasn't looking for a "how to fix" but more of a "what components would be good to look at"
18:30:48 mriedem mnaser: the computes are going to be running periodic tasks and pulling instances from the database over conductor
18:30:49 mnaser mriedem we toned down the service check ins (which is used by the services / scheduler) to 1 minute + 4 minute service_down_time and that helped
18:30:51 mriedem which is going to be rpc traffic
18:31:00 mriedem mnaser: this isn't the service check in
18:31:01 mnaser but yeah, periodic tasks is not something i've touched on yet
18:31:03 mriedem with the service group api
18:31:14 mriedem this would be things like the update_available_resources periodic task in the compute
18:31:36 mriedem that runs every minute by default to update resource usage information in the compute_nodes table and in the placement api
18:31:40 mriedem mnaser: which release?
18:31:53 mriedem there is also the instance_usage_audit task but that's disabled by default
18:31:59 mnaser i'll throw up debugging on a compute node
18:32:00 mnaser newton atm, no placement service in place yet
18:32:05 mriedem ok
18:32:10 mnaser yeah instance_usage_audit is off (well, i havent verified but notifications queue is empty and no consumers for it)
18:32:15 mriedem even w/o placement there is probably a lot of steady state traffic
18:32:35 mriedem the computes are also sending instance info to the filter scheduler every minute
18:32:37 mriedem *all* computes
18:32:42 mriedem even if you're not using the filter scheduler
18:32:45 mriedem you're welcome
18:32:53 mnaser lol!
18:33:12 mriedem are you using the filter scheduler or caching scheduler?
18:33:39 mnaser we are so i guess that helps, but 60 nodes every minute means 60 updates only so that averages to just one per second (and i think fuzzing in place in nova can help alleviate the big burst)
18:33:43 mnaser caching scheduler...? til.
18:33:56 mnaser the filter does the job for us though we had to reorder things because it took too long sometimes
18:34:13 mnaser aka computefilter is the last one but by the time it runs, "service is down"
18:34:50 mnaser (i dont know why it isn't earlier in by default, considering it's one of the more fundamental ones that might drop a whole bunch)
18:35:04 mriedem these are the current defaults
18:35:06 mriedem "ComputeCapabilitiesFilter",
18:35:06 mriedem "ComputeFilter",
18:35:06 mriedem "DiskFilter",
18:35:06 mriedem "RamFilter",
18:35:06 mriedem "AvailabilityZoneFilter",
18:35:06 mriedem "RetryFilter",
18:35:06 mriedem default=[
18:35:07 mriedem "ServerGroupAntiAffinityFilter",
18:35:07 mriedem "ImagePropertiesFilter",
18:35:08 mriedem ],
18:35:08 mriedem "ServerGroupAffinityFilter",
18:35:36 mriedem i think we can drop ramfilter and diskfilter from defaults now that we have placement required in ocata, but would have to check that out again
18:35:59 mnaser computefilter pretty much checks if service is alive
18:36:00 mnaser so... id imagine that's more important than figuring out if you have enough ram
18:36:22 mnaser but i guess the logic was ram will eliminate a bigger subset of possiblities rather than disabled nodes
18:36:31 mnaser but yeah, placement mix address a whole bunch of that
18:36:33 mriedem assuming there was logic
18:36:54 mriedem mnaser: anyway, let us know if you find anything
18:37:03 mriedem i always like to hear about performance issues
18:37:33 mnaser we've had some neutron ones but so far with nova, it's just trying to see how we can minimize rpc traffic so to handle big loads
18:37:51 mnaser we had the pleasure of an api request with num_instances => 200
18:38:31 mnaser nova seemed to cope well, but neutron couldn't catch up (more specifically neutron-dhcp-agent), so at some point instance port plugging started timing out (vif_plugging_timeout was getting hit)
18:40:28 mnaser which did raise the issue of.. there is no way of limiting num_instances in api requests (based on my research of the code base), so if the quota is 100 vms.. with one api request you can cause a lot of trouble (rate limiting wouldn't be a solution here considering its a single request)

Earlier   Later