Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-14
15:17:49 kashyap mriedem: Nice sleuthing on comment#16
15:18:03 kashyap How did you guess it could be related to AppArmour?
15:18:11 mriedem kashyap: i'm not sure that it is,
15:18:31 mriedem kashyap: but we've had those previous bugs where based on the version of libvirt, the script path in the interface xml has to be None or ''
15:18:44 mriedem to tell qemu use the default, which is /etc/qemu-ifup
15:18:51 mriedem and apparmor could be blocking that script
15:19:03 kashyap mriedem: Yeah, just checking with Laine Stump on OFTC network, who mediatates on libvirt networking
15:20:41 kashyap mriedem: Yeah, I recall the script path thing, that's fixed, no
15:20:42 kashyap ?
15:21:23 mriedem it is
15:21:47 mriedem kashyap: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L62
15:24:24 mriedem jaypipes: are you able to pull out of your bag of lore what the simple difference between disk_available_least and free_disk_gb is?
15:24:40 mriedem because the descriptions on those response parameters in https://developer.openstack.org/api-ref/compute/?expanded=show-hypervisor-statistics-detail#show-hypervisor-statistics is not clear
15:25:13 jaypipes mriedem: it's not clear because the implementation/design of it all is unclear...
15:25:13 kashyap mriedem: Laine & DanPB said: Without looking at data from the guest, it's hard to say *where* precisely is the delay lurking
15:25:25 mriedem jaypipes: plus it depends on backend storage doesn't it?
15:25:31 mriedem e.g. ceph vs others
15:26:01 jaypipes mriedem: disk_available_least is essentially the worst-case scenario whereas free_disk_gb considers sparse disks as not consuming their entire mapping file
15:26:21 kashyap mriedem: Are you able to reproduce this issue? You note a hit in the bug
15:26:22 jaypipes s/mapping/backing
15:26:34 mriedem kashyap: reproduce locally? i haven't tried.
15:26:43 kashyap mriedem: Okay, here's what the folks from #virt say:
15:26:46 mriedem kashyap: we had a recreate in upstream ci which is where i pulled and dug through those libvirtd logs
15:26:59 kashyap 17:19 < danpb> kashyap: suggests the guest OS is not honouring the unplug event promtly
15:27:02 kashyap 17:20 < danpb> kashyap: or perhaps it is honouring it, but something in the guest is waiting 5 minutes for something to happen before offlineing the nic
15:27:05 kashyap 17:20 < danpb> its hard to say for sure without getting data from the guest os really
15:27:08 kashyap 17:24 <@laine> kashyap: I was going to suggest what danpb suggested - the delay could be on the guest.
15:27:35 kashyap 17:26 < danpb> might need udev event logs for example
15:27:35 kashyap 17:25 < danpb> i'm not even sure how to get any data from the guest
15:27:52 kashyap mriedem: Does DevStack log 'udev' events? /me thinks no
15:28:05 mriedem kashyap: hmm, ok - is there some way to get information out of the guest, like the qemu guest agent? or should we dump the console when this times out?
15:28:14 mriedem kashyap: don't know
15:28:34 kashyap mriedem: Guest Agent is a good idea, but I'm not sure if it gives us what we need here. /me taps on the table and thinks
15:29:12 kashyap And looks at the man pages
15:30:43 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: cleanup PUT /os-hypervisors/statistics docs https://review.openstack.org/474250
15:47:59 openstackgerrit Merged openstack/nova master: Indicate Hyper-v supports fibre channel in support matrix https://review.openstack.org/473986
15:48:45 openstackgerrit Merged openstack/nova master: Remove service control from feature support matrix https://review.openstack.org/473989
15:54:45 jaypipes melwitt: I'm going to rebase 469635 and fix bauzas' comments from the dependent patch. ok with you?
15:54:58 melwitt jaypipes: I'm in the middle of doing it right now
15:55:10 jaypipes melwitt: ah, no probs!
15:55:16 melwitt thanks though
15:55:18 jaypipes :)
15:58:24 melwitt jaypipes: should I add the api-refs? that's the only thing I wasn't clear on
15:58:42 jaypipes melwitt: no, since the allocations api-ref is a dependency and we're still working on that
15:59:12 jaypipes melwitt: kinda pointless adding an api-ref about usages that discusses allocations and not have an api-ref on allocations :)
15:59:19 jaypipes which I mentioned to bauzas on 469635
15:59:22 melwitt agreed
15:59:49 jaypipes avolkov and jichenc are working on allocations api-ref
16:00:35 melwitt sweet
16:17:35 openstackgerrit Merged openstack/python-novaclient master: Clean up ShellTest unit tests https://review.openstack.org/473910
16:21:51 openstackgerrit Jay Pipes proposed openstack/nova master: Transform keypair.import notification https://review.openstack.org/467514
16:28:34 melwitt jaypipes: quick question, did you want to do something about the fetchall() bauzas mentioned? I addressed everything except that and the api-refs
16:28:57 melwitt in my local changes
16:30:59 melwitt jaypipes: I can just push what I have and if you want to do anything about the fetchall(), I'll leave it to you :)
16:31:28 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:31:29 jaypipes melwitt: meh? :)
16:31:29 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:42:40 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:42:41 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:43:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Only auto-disable new nova-compute services https://review.openstack.org/474285
17:07:19 cfriesen bauzas: you on?
17:11:50 cfriesen bauzas: (and any other cores) I'm running into an issue that prevents cold migrating an instance created originally on Liberty or earlier from a Mitaka compute node to a Newton compute node. It appears that until you run "nova db online_data_migrations" to create the missing RequestSpec you can't migrate. Traceback at http://paste.openstack.org/show/612574/
17:12:23 cfriesen Is this design intent? If so, it would seem to preclude an upgrade involving an underlying OS upgrade at the same time as an openstack upgrade.
17:13:16 cfriesen (note: some of the line numbers might not quite match up with upstream newton since we've patched some things)
17:13:47 cfriesen I'm confirming vanilla stable/newton now
17:23:14 mriedem melwitt: ah i see now, objects.Quotas._remove_members_in_db - yeah that won't work :)
17:24:42 melwitt mriedem: yup. thanks for pointing out the lack of tests. I added similar tests to all the "Count <resource> to check quota" patches except instances, that one is in progress
17:27:12 cfriesen mriedem: do you know whether the above issue ^ is design intent? It seems like an oversight. Basically we call objects.RequestSpec.save() on a brand new object, which calls _save_in_db(), which calls RequestSpec._get_by_instance_uuid_from_db(), which raises exception.RequestSpecNotFound()
17:27:55 mriedem cfriesen: would have to dig, but bauzas is the best answer for that, but he's on PTO today
17:27:56 mriedem back tomorrow
17:28:05 mriedem and i can't dig right now
17:28:47 cfriesen generally speaking it seems like the newton code is assuming that we've already run "nova db online_data_migrations", but we can't run that until all the nodes are running newton
17:28:52 dansmith cfriesen: why do you think it's an oversight? the newer nodes expect request specs to be in place, the online data migration corrects them for older instances
17:29:01 dansmith cfriesen: that's not true,
17:29:20 cfriesen dansmith: what's not true?
17:29:35 dansmith you can run it right after you've upgraded your schema, after control services are upgraded
17:29:38 dansmith can/should
17:29:50 dansmith and anything that can't run because computes aren't upgraded won't, like that pci one
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: Make compute_node_statistics() work across cells https://review.openstack.org/473931
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565
17:31:29 cfriesen dansmith: https://docs.openstack.org/developer/nova/upgrade.html says to run it "once all services are running the new code"
17:32:05 cfriesen dansmith: or at least in the "after maintenance window" section along with other stuff that runs after all services are upgraded
17:32:49 cfriesen dansmith: so you're saying to run it once after control services are upgraded and then again after everything is upgraded?
17:32:51 dansmith yeah, I see the confusion,
17:32:56 dansmith what that's describing is the maintenance window for your control services, which requires total downtime,
17:33:01 dansmith after which you can roll your computes later
17:33:53 dansmith although it's also confusing the situation with the config bits being before that
17:33:54 cfriesen dansmith: it can't be the maintenance window for just control services, it talks about bumping the RPC version
17:34:10 cfriesen "Once all the services are running the latest version of the code, and all the services are aware they all have been upgraded, it is safe to transform the data in the database into its new format."
17:34:12 dansmith yeah, its' confusing
17:34:57 dansmith normally you wouldn't really have to run it twice, and either after control or after everything is done would be enough,
17:35:14 dansmith but if you've got a release where there are critical things on both sides...
17:35:38 cfriesen dansmith: do we guarantee that the online data migrations won't change things in such a way that they break old compute nodes?
17:35:50 dansmith that's the whole point of conductor
17:35:54 dansmith to insulate computes from those changes
17:40:40 cfriesen dansmith: okay. that answers my question, but I think we need to update the docs at https://docs.openstack.org/developer/nova/upgrade.html to make it clear that the online data migrations should be run earlier *during* the maintenance window.
17:41:09 cfriesen dansmith: and remove the comments that talk about it only needing to be run before the upgrade to the *next* version
17:41:43 dansmith cfriesen: perhaps mentioning that if you fail to reduce the remaining updates past a certain threshold, those may be things that need to run again later
17:42:52 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
17:42:53 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635

Earlier   Later