Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-14
16:21:51 openstackgerrit Jay Pipes proposed openstack/nova master: Transform keypair.import notification https://review.openstack.org/467514
16:28:34 melwitt jaypipes: quick question, did you want to do something about the fetchall() bauzas mentioned? I addressed everything except that and the api-refs
16:28:57 melwitt in my local changes
16:30:59 melwitt jaypipes: I can just push what I have and if you want to do anything about the fetchall(), I'll leave it to you :)
16:31:28 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:31:29 jaypipes melwitt: meh? :)
16:31:29 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:42:40 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:42:41 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:43:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Only auto-disable new nova-compute services https://review.openstack.org/474285
17:07:19 cfriesen bauzas: you on?
17:11:50 cfriesen bauzas: (and any other cores) I'm running into an issue that prevents cold migrating an instance created originally on Liberty or earlier from a Mitaka compute node to a Newton compute node. It appears that until you run "nova db online_data_migrations" to create the missing RequestSpec you can't migrate. Traceback at http://paste.openstack.org/show/612574/
17:12:23 cfriesen Is this design intent? If so, it would seem to preclude an upgrade involving an underlying OS upgrade at the same time as an openstack upgrade.
17:13:16 cfriesen (note: some of the line numbers might not quite match up with upstream newton since we've patched some things)
17:13:47 cfriesen I'm confirming vanilla stable/newton now
17:23:14 mriedem melwitt: ah i see now, objects.Quotas._remove_members_in_db - yeah that won't work :)
17:24:42 melwitt mriedem: yup. thanks for pointing out the lack of tests. I added similar tests to all the "Count <resource> to check quota" patches except instances, that one is in progress
17:27:12 cfriesen mriedem: do you know whether the above issue ^ is design intent? It seems like an oversight. Basically we call objects.RequestSpec.save() on a brand new object, which calls _save_in_db(), which calls RequestSpec._get_by_instance_uuid_from_db(), which raises exception.RequestSpecNotFound()
17:27:55 mriedem cfriesen: would have to dig, but bauzas is the best answer for that, but he's on PTO today
17:27:56 mriedem back tomorrow
17:28:05 mriedem and i can't dig right now
17:28:47 cfriesen generally speaking it seems like the newton code is assuming that we've already run "nova db online_data_migrations", but we can't run that until all the nodes are running newton
17:28:52 dansmith cfriesen: why do you think it's an oversight? the newer nodes expect request specs to be in place, the online data migration corrects them for older instances
17:29:01 dansmith cfriesen: that's not true,
17:29:20 cfriesen dansmith: what's not true?
17:29:35 dansmith you can run it right after you've upgraded your schema, after control services are upgraded
17:29:38 dansmith can/should
17:29:50 dansmith and anything that can't run because computes aren't upgraded won't, like that pci one
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: Make compute_node_statistics() work across cells https://review.openstack.org/473931
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565
17:31:29 cfriesen dansmith: https://docs.openstack.org/developer/nova/upgrade.html says to run it "once all services are running the new code"
17:32:05 cfriesen dansmith: or at least in the "after maintenance window" section along with other stuff that runs after all services are upgraded
17:32:49 cfriesen dansmith: so you're saying to run it once after control services are upgraded and then again after everything is upgraded?
17:32:51 dansmith yeah, I see the confusion,
17:32:56 dansmith what that's describing is the maintenance window for your control services, which requires total downtime,
17:33:01 dansmith after which you can roll your computes later
17:33:53 dansmith although it's also confusing the situation with the config bits being before that
17:33:54 cfriesen dansmith: it can't be the maintenance window for just control services, it talks about bumping the RPC version
17:34:10 cfriesen "Once all the services are running the latest version of the code, and all the services are aware they all have been upgraded, it is safe to transform the data in the database into its new format."
17:34:12 dansmith yeah, its' confusing
17:34:57 dansmith normally you wouldn't really have to run it twice, and either after control or after everything is done would be enough,
17:35:14 dansmith but if you've got a release where there are critical things on both sides...
17:35:38 cfriesen dansmith: do we guarantee that the online data migrations won't change things in such a way that they break old compute nodes?
17:35:50 dansmith that's the whole point of conductor
17:35:54 dansmith to insulate computes from those changes
17:40:40 cfriesen dansmith: okay. that answers my question, but I think we need to update the docs at https://docs.openstack.org/developer/nova/upgrade.html to make it clear that the online data migrations should be run earlier *during* the maintenance window.
17:41:09 cfriesen dansmith: and remove the comments that talk about it only needing to be run before the upgrade to the *next* version
17:41:43 dansmith cfriesen: perhaps mentioning that if you fail to reduce the remaining updates past a certain threshold, those may be things that need to run again later
17:42:52 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
17:42:53 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
17:42:58 dansmith I probably wrote that text and was writing it attempting to hit a generic scenario without mentioning every possibility, where I just put it in the process somewhere
17:42:58 cfriesen dansmith: doesn't anything greater than zero mean you need to re-run it later?
17:43:14 dansmith cfriesen: if you run with --until-done or whatever, yeah
17:43:44 cfriesen dansmith: originally I think the idea was that you'd only need to run the online_data_migrations sometime before the next upgrade...that's the way it's written at https://docs.openstack.org/releasenotes/nova/newton.html
17:43:57 cfriesen "... budget time to run these online migrations before you plan to upgrade to ocata"
17:44:02 dansmith cfriesen: that has never been the "design intent"
17:44:32 dansmith because as those docs say, you're expecting more conductor load during the period where you're not fully migrated, so budget resources for that
17:45:29 cfriesen dansmith: there's been a disconnect then, because that sort of wording is all over the upstream documentation around upgrades
17:46:09 dansmith cfriesen: patches welcome
17:50:09 cfriesen dansmith: maybe next week. I'm now going to be busy modifying our automated upgrade tools and retesting everything.
17:56:01 cfriesen this won't impact people that upgrade nova-compute with all the guests still running....it's only a factor if you want to migrate all the instances off the "old" compute node to upgrade the OS as well.
17:59:51 cfriesen dansmith: question for you....if some online migrations can't complete until all the services are upgraded (like pcidevice_online_data_migration()) and others *must* complete in order to allow things to function, how does the person running the command know if it's safe to proceed?
18:01:15 cfriesen dansmith: it makes it really hard to reliably automate the call to "nova-manage db online_data_migrations"
18:02:02 cfriesen seems to me we need to break it up into two calls, one right after the control services are upgraded, and one after all the compute nodes are also upgraded
18:03:17 dansmith we've only ever had that one thing that needed to run after computes are up, and it's gone now
18:03:41 dansmith it's a very exceptional case and once you did it on any release it was present on, it was done
18:03:53 dansmith so not in favor of complicating things further based on one exceptional example
18:04:05 dansmith it could have printed a warning explanation and been fine I think
18:04:21 dansmith "sorry homey, I can't finish until after, so blah blah"
18:04:22 cfriesen dansmith: okay, didn't realize that was the only thing ever
18:04:52 dansmith yep, the only one ever, which is why in my head, you do this when control is upgraded
18:21:29 mnaser i'm trying to do some perf tweaking and i'm just curious on why a relatively idle 60-ish node cloud conductor rabbitmq message rates is at ~20/s
18:21:52 mnaser i increased the service report interval from 10 seconds up to 60s
18:22:13 mnaser and why i say relatively idle, there is no new servers being added/deleted etc
18:27:15 mriedem melwitt: one small thing in here https://review.openstack.org/#/c/446241/
18:27:26 mriedem melwitt: in a normal network delete, we call network_api.disassociate and then delete,
18:27:32 mriedem so we should do the same for that rollback
18:27:43 mriedem melwitt: i also found a fun latent networks quota bug in there
18:28:56 mriedem mnaser: could be a few things, periodic tasks in the computes come to mine
18:28:58 mriedem *mind
18:29:02 melwitt mriedem: cool, thanks. looking
18:29:14 mriedem might also depend on if you have instance_usage_audit enabled to send notifications
18:29:39 edleafe mnaser: You should probably ask on the #openstack channel. See #topic
18:30:02 mriedem edleafe: i actually want to know what mnaser is doing and finding out, given he runs vexxhost :)
18:30:09 mnaser edleafe im aware, i was just looking more on the internals of nova, i wasn't looking for a "how to fix" but more of a "what components would be good to look at"
18:30:48 mriedem mnaser: the computes are going to be running periodic tasks and pulling instances from the database over conductor
18:30:49 mnaser mriedem we toned down the service check ins (which is used by the services / scheduler) to 1 minute + 4 minute service_down_time and that helped
18:30:51 mriedem which is going to be rpc traffic
18:31:00 mriedem mnaser: this isn't the service check in
18:31:01 mnaser but yeah, periodic tasks is not something i've touched on yet
18:31:03 mriedem with the service group api
18:31:14 mriedem this would be things like the update_available_resources periodic task in the compute
18:31:36 mriedem that runs every minute by default to update resource usage information in the compute_nodes table and in the placement api
18:31:40 mriedem mnaser: which release?
18:31:53 mriedem there is also the instance_usage_audit task but that's disabled by default
18:31:59 mnaser i'll throw up debugging on a compute node
18:32:00 mnaser newton atm, no placement service in place yet
18:32:05 mriedem ok
18:32:10 mnaser yeah instance_usage_audit is off (well, i havent verified but notifications queue is empty and no consumers for it)
18:32:15 mriedem even w/o placement there is probably a lot of steady state traffic

Earlier   Later