Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-14
15:26:43 kashyap mriedem: Okay, here's what the folks from #virt say:
15:26:46 mriedem kashyap: we had a recreate in upstream ci which is where i pulled and dug through those libvirtd logs
15:26:59 kashyap 17:19 < danpb> kashyap: suggests the guest OS is not honouring the unplug event promtly
15:27:02 kashyap 17:20 < danpb> kashyap: or perhaps it is honouring it, but something in the guest is waiting 5 minutes for something to happen before offlineing the nic
15:27:05 kashyap 17:20 < danpb> its hard to say for sure without getting data from the guest os really
15:27:08 kashyap 17:24 <@laine> kashyap: I was going to suggest what danpb suggested - the delay could be on the guest.
15:27:35 kashyap 17:26 < danpb> might need udev event logs for example
15:27:35 kashyap 17:25 < danpb> i'm not even sure how to get any data from the guest
15:27:52 kashyap mriedem: Does DevStack log 'udev' events? /me thinks no
15:28:05 mriedem kashyap: hmm, ok - is there some way to get information out of the guest, like the qemu guest agent? or should we dump the console when this times out?
15:28:14 mriedem kashyap: don't know
15:28:34 kashyap mriedem: Guest Agent is a good idea, but I'm not sure if it gives us what we need here. /me taps on the table and thinks
15:29:12 kashyap And looks at the man pages
15:30:43 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: cleanup PUT /os-hypervisors/statistics docs https://review.openstack.org/474250
15:47:59 openstackgerrit Merged openstack/nova master: Indicate Hyper-v supports fibre channel in support matrix https://review.openstack.org/473986
15:48:45 openstackgerrit Merged openstack/nova master: Remove service control from feature support matrix https://review.openstack.org/473989
15:54:45 jaypipes melwitt: I'm going to rebase 469635 and fix bauzas' comments from the dependent patch. ok with you?
15:54:58 melwitt jaypipes: I'm in the middle of doing it right now
15:55:10 jaypipes melwitt: ah, no probs!
15:55:16 melwitt thanks though
15:55:18 jaypipes :)
15:58:24 melwitt jaypipes: should I add the api-refs? that's the only thing I wasn't clear on
15:58:42 jaypipes melwitt: no, since the allocations api-ref is a dependency and we're still working on that
15:59:12 jaypipes melwitt: kinda pointless adding an api-ref about usages that discusses allocations and not have an api-ref on allocations :)
15:59:19 jaypipes which I mentioned to bauzas on 469635
15:59:22 melwitt agreed
15:59:49 jaypipes avolkov and jichenc are working on allocations api-ref
16:00:35 melwitt sweet
16:17:35 openstackgerrit Merged openstack/python-novaclient master: Clean up ShellTest unit tests https://review.openstack.org/473910
16:21:51 openstackgerrit Jay Pipes proposed openstack/nova master: Transform keypair.import notification https://review.openstack.org/467514
16:28:34 melwitt jaypipes: quick question, did you want to do something about the fetchall() bauzas mentioned? I addressed everything except that and the api-refs
16:28:57 melwitt in my local changes
16:30:59 melwitt jaypipes: I can just push what I have and if you want to do anything about the fetchall(), I'll leave it to you :)
16:31:28 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:31:29 jaypipes melwitt: meh? :)
16:31:29 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:42:40 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:42:41 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:43:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Only auto-disable new nova-compute services https://review.openstack.org/474285
17:07:19 cfriesen bauzas: you on?
17:11:50 cfriesen bauzas: (and any other cores) I'm running into an issue that prevents cold migrating an instance created originally on Liberty or earlier from a Mitaka compute node to a Newton compute node. It appears that until you run "nova db online_data_migrations" to create the missing RequestSpec you can't migrate. Traceback at http://paste.openstack.org/show/612574/
17:12:23 cfriesen Is this design intent? If so, it would seem to preclude an upgrade involving an underlying OS upgrade at the same time as an openstack upgrade.
17:13:16 cfriesen (note: some of the line numbers might not quite match up with upstream newton since we've patched some things)
17:13:47 cfriesen I'm confirming vanilla stable/newton now
17:23:14 mriedem melwitt: ah i see now, objects.Quotas._remove_members_in_db - yeah that won't work :)
17:24:42 melwitt mriedem: yup. thanks for pointing out the lack of tests. I added similar tests to all the "Count <resource> to check quota" patches except instances, that one is in progress
17:27:12 cfriesen mriedem: do you know whether the above issue ^ is design intent? It seems like an oversight. Basically we call objects.RequestSpec.save() on a brand new object, which calls _save_in_db(), which calls RequestSpec._get_by_instance_uuid_from_db(), which raises exception.RequestSpecNotFound()
17:27:55 mriedem cfriesen: would have to dig, but bauzas is the best answer for that, but he's on PTO today
17:27:56 mriedem back tomorrow
17:28:05 mriedem and i can't dig right now
17:28:47 cfriesen generally speaking it seems like the newton code is assuming that we've already run "nova db online_data_migrations", but we can't run that until all the nodes are running newton
17:28:52 dansmith cfriesen: why do you think it's an oversight? the newer nodes expect request specs to be in place, the online data migration corrects them for older instances
17:29:01 dansmith cfriesen: that's not true,
17:29:20 cfriesen dansmith: what's not true?
17:29:35 dansmith you can run it right after you've upgraded your schema, after control services are upgraded
17:29:38 dansmith can/should
17:29:50 dansmith and anything that can't run because computes aren't upgraded won't, like that pci one
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: Make compute_node_statistics() work across cells https://review.openstack.org/473931
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565
17:31:29 cfriesen dansmith: https://docs.openstack.org/developer/nova/upgrade.html says to run it "once all services are running the new code"
17:32:05 cfriesen dansmith: or at least in the "after maintenance window" section along with other stuff that runs after all services are upgraded
17:32:49 cfriesen dansmith: so you're saying to run it once after control services are upgraded and then again after everything is upgraded?
17:32:51 dansmith yeah, I see the confusion,
17:32:56 dansmith what that's describing is the maintenance window for your control services, which requires total downtime,
17:33:01 dansmith after which you can roll your computes later
17:33:53 dansmith although it's also confusing the situation with the config bits being before that
17:33:54 cfriesen dansmith: it can't be the maintenance window for just control services, it talks about bumping the RPC version
17:34:10 cfriesen "Once all the services are running the latest version of the code, and all the services are aware they all have been upgraded, it is safe to transform the data in the database into its new format."
17:34:12 dansmith yeah, its' confusing
17:34:57 dansmith normally you wouldn't really have to run it twice, and either after control or after everything is done would be enough,
17:35:14 dansmith but if you've got a release where there are critical things on both sides...
17:35:38 cfriesen dansmith: do we guarantee that the online data migrations won't change things in such a way that they break old compute nodes?
17:35:50 dansmith that's the whole point of conductor
17:35:54 dansmith to insulate computes from those changes
17:40:40 cfriesen dansmith: okay. that answers my question, but I think we need to update the docs at https://docs.openstack.org/developer/nova/upgrade.html to make it clear that the online data migrations should be run earlier *during* the maintenance window.
17:41:09 cfriesen dansmith: and remove the comments that talk about it only needing to be run before the upgrade to the *next* version
17:41:43 dansmith cfriesen: perhaps mentioning that if you fail to reduce the remaining updates past a certain threshold, those may be things that need to run again later
17:42:52 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
17:42:53 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
17:42:58 dansmith I probably wrote that text and was writing it attempting to hit a generic scenario without mentioning every possibility, where I just put it in the process somewhere
17:42:58 cfriesen dansmith: doesn't anything greater than zero mean you need to re-run it later?
17:43:14 dansmith cfriesen: if you run with --until-done or whatever, yeah
17:43:44 cfriesen dansmith: originally I think the idea was that you'd only need to run the online_data_migrations sometime before the next upgrade...that's the way it's written at https://docs.openstack.org/releasenotes/nova/newton.html
17:43:57 cfriesen "... budget time to run these online migrations before you plan to upgrade to ocata"
17:44:02 dansmith cfriesen: that has never been the "design intent"
17:44:32 dansmith because as those docs say, you're expecting more conductor load during the period where you're not fully migrated, so budget resources for that
17:45:29 cfriesen dansmith: there's been a disconnect then, because that sort of wording is all over the upstream documentation around upgrades
17:46:09 dansmith cfriesen: patches welcome
17:50:09 cfriesen dansmith: maybe next week. I'm now going to be busy modifying our automated upgrade tools and retesting everything.
17:56:01 cfriesen this won't impact people that upgrade nova-compute with all the guests still running....it's only a factor if you want to migrate all the instances off the "old" compute node to upgrade the OS as well.
17:59:51 cfriesen dansmith: question for you....if some online migrations can't complete until all the services are upgraded (like pcidevice_online_data_migration()) and others *must* complete in order to allow things to function, how does the person running the command know if it's safe to proceed?
18:01:15 cfriesen dansmith: it makes it really hard to reliably automate the call to "nova-manage db online_data_migrations"
18:02:02 cfriesen seems to me we need to break it up into two calls, one right after the control services are upgraded, and one after all the compute nodes are also upgraded
18:03:17 dansmith we've only ever had that one thing that needed to run after computes are up, and it's gone now
18:03:41 dansmith it's a very exceptional case and once you did it on any release it was present on, it was done
18:03:53 dansmith so not in favor of complicating things further based on one exceptional example
18:04:05 dansmith it could have printed a warning explanation and been fine I think
18:04:21 dansmith "sorry homey, I can't finish until after, so blah blah"
18:04:22 cfriesen dansmith: okay, didn't realize that was the only thing ever
18:04:52 dansmith yep, the only one ever, which is why in my head, you do this when control is upgraded

Earlier   Later