Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-14
15:04:16 dansmith actually
15:04:33 dansmith we should probably only run this if we're on a node where there is something like nova-api configured, so I'll do that
15:05:15 mriedem dansmith: that's what devstack/stack.sh checks for
15:05:17 mriedem so yeah
15:05:26 dansmith yeah
15:05:54 gibi mriedem: I have to run now, but will get back to the snapshot patch tomorrow morning. thanks again for the review!
15:06:18 mriedem yw
15:07:40 mriedem man we don't even go over conductor for compute_node_statistics
15:08:22 dansmith mriedem: well, it's api only, so we wouldn't
15:08:24 dansmith but no objects yeah
15:08:52 dtantsur dansmith: seems so: https://github.com/openstack/ironic/blob/870d909cc7e0c6118a174d412ae814953bb805e1/devstack/lib/ironic#L377-L384
15:09:16 dansmith dtantsur: I think checking for n-api is the devstacky way
15:09:24 dansmith dtantsur: just pushed a rev for that
15:09:40 dtantsur thanks, I'll check after a meeting (and probably after Jenkins agrees with it)
15:09:44 dansmith cool, thanks
15:09:56 dansmith I'll be keeping an eye on it too of course :)
15:10:12 dansmith glad we (mriedem) caught this before we landed it and broke ironic this time :)
15:10:29 mriedem \o/
15:10:32 dansmith heh
15:17:49 kashyap mriedem: Nice sleuthing on comment#16
15:18:03 kashyap How did you guess it could be related to AppArmour?
15:18:11 mriedem kashyap: i'm not sure that it is,
15:18:31 mriedem kashyap: but we've had those previous bugs where based on the version of libvirt, the script path in the interface xml has to be None or ''
15:18:44 mriedem to tell qemu use the default, which is /etc/qemu-ifup
15:18:51 mriedem and apparmor could be blocking that script
15:19:03 kashyap mriedem: Yeah, just checking with Laine Stump on OFTC network, who mediatates on libvirt networking
15:20:41 kashyap mriedem: Yeah, I recall the script path thing, that's fixed, no
15:20:42 kashyap ?
15:21:23 mriedem it is
15:21:47 mriedem kashyap: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L62
15:24:24 mriedem jaypipes: are you able to pull out of your bag of lore what the simple difference between disk_available_least and free_disk_gb is?
15:24:40 mriedem because the descriptions on those response parameters in https://developer.openstack.org/api-ref/compute/?expanded=show-hypervisor-statistics-detail#show-hypervisor-statistics is not clear
15:25:13 jaypipes mriedem: it's not clear because the implementation/design of it all is unclear...
15:25:13 kashyap mriedem: Laine & DanPB said: Without looking at data from the guest, it's hard to say *where* precisely is the delay lurking
15:25:25 mriedem jaypipes: plus it depends on backend storage doesn't it?
15:25:31 mriedem e.g. ceph vs others
15:26:01 jaypipes mriedem: disk_available_least is essentially the worst-case scenario whereas free_disk_gb considers sparse disks as not consuming their entire mapping file
15:26:21 kashyap mriedem: Are you able to reproduce this issue? You note a hit in the bug
15:26:22 jaypipes s/mapping/backing
15:26:34 mriedem kashyap: reproduce locally? i haven't tried.
15:26:43 kashyap mriedem: Okay, here's what the folks from #virt say:
15:26:46 mriedem kashyap: we had a recreate in upstream ci which is where i pulled and dug through those libvirtd logs
15:26:59 kashyap 17:19 < danpb> kashyap: suggests the guest OS is not honouring the unplug event promtly
15:27:02 kashyap 17:20 < danpb> kashyap: or perhaps it is honouring it, but something in the guest is waiting 5 minutes for something to happen before offlineing the nic
15:27:05 kashyap 17:20 < danpb> its hard to say for sure without getting data from the guest os really
15:27:08 kashyap 17:24 <@laine> kashyap: I was going to suggest what danpb suggested - the delay could be on the guest.
15:27:35 kashyap 17:26 < danpb> might need udev event logs for example
15:27:35 kashyap 17:25 < danpb> i'm not even sure how to get any data from the guest
15:27:52 kashyap mriedem: Does DevStack log 'udev' events? /me thinks no
15:28:05 mriedem kashyap: hmm, ok - is there some way to get information out of the guest, like the qemu guest agent? or should we dump the console when this times out?
15:28:14 mriedem kashyap: don't know
15:28:34 kashyap mriedem: Guest Agent is a good idea, but I'm not sure if it gives us what we need here. /me taps on the table and thinks
15:29:12 kashyap And looks at the man pages
15:30:43 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: cleanup PUT /os-hypervisors/statistics docs https://review.openstack.org/474250
15:47:59 openstackgerrit Merged openstack/nova master: Indicate Hyper-v supports fibre channel in support matrix https://review.openstack.org/473986
15:48:45 openstackgerrit Merged openstack/nova master: Remove service control from feature support matrix https://review.openstack.org/473989
15:54:45 jaypipes melwitt: I'm going to rebase 469635 and fix bauzas' comments from the dependent patch. ok with you?
15:54:58 melwitt jaypipes: I'm in the middle of doing it right now
15:55:10 jaypipes melwitt: ah, no probs!
15:55:16 melwitt thanks though
15:55:18 jaypipes :)
15:58:24 melwitt jaypipes: should I add the api-refs? that's the only thing I wasn't clear on
15:58:42 jaypipes melwitt: no, since the allocations api-ref is a dependency and we're still working on that
15:59:12 jaypipes melwitt: kinda pointless adding an api-ref about usages that discusses allocations and not have an api-ref on allocations :)
15:59:19 jaypipes which I mentioned to bauzas on 469635
15:59:22 melwitt agreed
15:59:49 jaypipes avolkov and jichenc are working on allocations api-ref
16:00:35 melwitt sweet
16:17:35 openstackgerrit Merged openstack/python-novaclient master: Clean up ShellTest unit tests https://review.openstack.org/473910
16:21:51 openstackgerrit Jay Pipes proposed openstack/nova master: Transform keypair.import notification https://review.openstack.org/467514
16:28:34 melwitt jaypipes: quick question, did you want to do something about the fetchall() bauzas mentioned? I addressed everything except that and the api-refs
16:28:57 melwitt in my local changes
16:30:59 melwitt jaypipes: I can just push what I have and if you want to do anything about the fetchall(), I'll leave it to you :)
16:31:28 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:31:29 jaypipes melwitt: meh? :)
16:31:29 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:42:40 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
16:42:41 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
16:43:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Only auto-disable new nova-compute services https://review.openstack.org/474285
17:07:19 cfriesen bauzas: you on?
17:11:50 cfriesen bauzas: (and any other cores) I'm running into an issue that prevents cold migrating an instance created originally on Liberty or earlier from a Mitaka compute node to a Newton compute node. It appears that until you run "nova db online_data_migrations" to create the missing RequestSpec you can't migrate. Traceback at http://paste.openstack.org/show/612574/
17:12:23 cfriesen Is this design intent? If so, it would seem to preclude an upgrade involving an underlying OS upgrade at the same time as an openstack upgrade.
17:13:16 cfriesen (note: some of the line numbers might not quite match up with upstream newton since we've patched some things)
17:13:47 cfriesen I'm confirming vanilla stable/newton now
17:23:14 mriedem melwitt: ah i see now, objects.Quotas._remove_members_in_db - yeah that won't work :)
17:24:42 melwitt mriedem: yup. thanks for pointing out the lack of tests. I added similar tests to all the "Count <resource> to check quota" patches except instances, that one is in progress
17:27:12 cfriesen mriedem: do you know whether the above issue ^ is design intent? It seems like an oversight. Basically we call objects.RequestSpec.save() on a brand new object, which calls _save_in_db(), which calls RequestSpec._get_by_instance_uuid_from_db(), which raises exception.RequestSpecNotFound()
17:27:55 mriedem cfriesen: would have to dig, but bauzas is the best answer for that, but he's on PTO today
17:27:56 mriedem back tomorrow
17:28:05 mriedem and i can't dig right now
17:28:47 cfriesen generally speaking it seems like the newton code is assuming that we've already run "nova db online_data_migrations", but we can't run that until all the nodes are running newton
17:28:52 dansmith cfriesen: why do you think it's an oversight? the newer nodes expect request specs to be in place, the online data migration corrects them for older instances
17:29:01 dansmith cfriesen: that's not true,
17:29:20 cfriesen dansmith: what's not true?
17:29:35 dansmith you can run it right after you've upgraded your schema, after control services are upgraded
17:29:38 dansmith can/should
17:29:50 dansmith and anything that can't run because computes aren't upgraded won't, like that pci one
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: Make compute_node_statistics() work across cells https://review.openstack.org/473931
17:30:28 openstackgerrit Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565
17:31:29 cfriesen dansmith: https://docs.openstack.org/developer/nova/upgrade.html says to run it "once all services are running the new code"

Earlier   Later