Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-14
16:32:09 melwitt gibi: that's good, I think you will get to the bottom of it :)
16:32:20 mriedem well if you want to get really depressed http://logs.openstack.org/47/635147/11/check/nova-tox-functional-py35/5c3c2a9/job-output.txt.gz#_2019-02-13_13_33_41_103932
16:32:34 mriedem that traceback wins the grand prize
16:32:41 mriedem and i don't know what to do about it
16:32:48 mriedem looks like the db dropped in the middle of the test
16:33:32 gibi mriedem: nice :)
16:33:56 mriedem the dumping of os-traits contents to the logs on each functional test run with the PlacementFixture isn't helping either
16:34:08 mriedem http://logs.openstack.org/47/635147/11/check/nova-tox-functional-py35/5c3c2a9/job-output.txt.gz#_2019-02-13_13_33_41_193979
16:34:24 gibi mriedem: I think we can move that to DEBUG
16:35:18 mriedem yeah probably http://logs.openstack.org/59/636459/2/check/tempest-full/3226eb1/controller/logs/screen-placement-api.txt.gz#_Feb_13_13_54_30_464250
16:35:23 gibi I'm affraid that the drop of the databease could be another interfering test case. Did you managed to reproduce that locally?
16:35:31 mriedem nope
16:35:35 bauzas gibi: efried: what's the best way to copy a providertree object ? copy.deepcopy() ?
16:35:54 openstackgerrit Elod Illes proposed openstack/nova master: DNM: test pip 19 - pip freeze with git editable https://review.openstack.org/636980
16:35:56 bauzas or do we have other built-in mechanism ?
16:36:18 bauzas tbc, I want to have a separate reference to be stored on the libvirt object
16:36:54 gibi bauzas: I would be affraid of copying the self.lock = lockutils.internal_lock(_LOCK_NAME) inside the ProviderTree
16:37:25 efried bauzas: upt is already operating on a copy... but I'm actually not positive that ufpt doesn't dork with it. So yeah, copy.deepcopy is probably the best bet.
16:37:58 gibi bauzas: besides the self.lock the rest of the internals seems copiable
16:38:18 bauzas gibi: so, ideally we should provide a copy method
16:38:19 efried gibi: hm, interesting point about the lock. This impl that bauzas is doing should be read-only. But that still wouldn't stop you from bouncing off the lock if the other instance is using it.
16:38:38 bauzas efried: what I want is purely read-only
16:38:45 efried However, I would be surprised if the lockutils lock doesn't override __copy__ to *not* copy itself.
16:38:49 bauzas actually, you know what ? I'm gonna make it explicit
16:38:55 bauzas by using a propertgy
16:39:11 bauzas we're not intended to manipulate this copy
16:39:12 gibi mriedem: I think I will stick to the remove-the-assert-that-blow-the-world issue, that I can reproduce locally at least
16:39:59 efried bauzas: There are methods to get a read-only copy of the data out of ProviderTree, but I'm pretty sure they only get one provider at a time.
16:40:22 efried Yeah, ProviderTree.data()
16:40:27 mriedem gibi: efried: fwiw i'm also holding off on the bw provider series until https://review.openstack.org/#/c/616239/ makes progress
16:40:40 mriedem and we've got 3 weeks from FF today
16:40:52 efried I guess that's my action
16:41:16 mriedem i'm just waiting for efried to +2 so i can nit pick a few things and make it look like i get it
16:41:17 gibi efried: thanks
16:41:21 bauzas efried: anyway, I'm rushing to propose a new rev
16:41:35 bauzas efried: you will be able to comment on the best approach by reviewing
16:41:41 bauzas not a big deal
16:41:50 gibi efried: to your surprise there is no copy overried in https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py ;)
16:41:57 efried gibi: d'oh!
16:44:11 efried uh, wtf. gibi, bauzas, when I try to deepcopy an object with a lockutils.internal_lock in it, I get an error.
16:44:35 efried so how tf would that be working in the rt? /me looks...
16:51:02 efried gibi, bauzas: http://paste.openstack.org/show/745111/ <== so how tf is https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L885 working??
16:51:37 bauzas heh
16:51:47 bauzas sorcery !
16:54:03 bauzas gibi: looking at the func test
16:54:05 bauzas gibi: you lost me
16:54:47 bauzas gibi: in theory, if you request a VGPU but we don't have inventories because the conf option is unset, you should end up with a beautiful NoValidHost
16:54:56 bauzas so I wonder why it works
16:55:08 bauzas oh wait
16:55:22 bauzas I missed the point you're creating the inventory manually
16:55:44 bauzas oki doki
16:55:58 bauzas so you do this to fake the fact you were running an old version
16:56:22 gibi bauzas: yeah, you are correct
16:56:51 bauzas ok, the good news is that I saw this when I looked at the provider_tree for the allocation RP UUID
16:57:02 bauzas and I was wondering why I was getting a root RP
16:57:10 bauzas gotcha
16:57:15 bauzas perfect, it works then
17:00:40 efried jaypipes: maybe you have some insight?
17:00:59 efried jaypipes: on: http://paste.openstack.org/show/745111/ <== so how tf is https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L885 working??
17:01:54 jaypipes looking...
17:04:09 gibi efried: sorry I'm expired for today
17:04:39 efried gibi: I think you've done enough, don't you? :P
17:05:06 efried "Ohh, I'm gibi, I notice things like lock copies, nya nya nya"
17:05:20 gibi :P
17:05:23 openstackgerrit Merged openstack/nova master: Use math.gcd starting with python 3.5 https://review.openstack.org/636669
17:05:35 openstackgerrit Merged openstack/nova master: doc: specify --os-compute-api-version when setting flavor description https://review.openstack.org/635508
17:05:46 openstackgerrit Merged openstack/nova master: Remove PLACEMENT_DB_ENABLED from nova-next job config https://review.openstack.org/634953
17:10:29 jaypipes efried: I have no idea how that works.
17:11:48 efried jaypipes: I figured we must be monkey-patching lockutils somewhere. Couldn't find it at a glance, but haven't finished looking.
17:12:02 jaypipes no idea. :(
17:13:16 openstackgerrit Stephen Finucane proposed openstack/os-vif master: WIP: Add API docs for various VIF types https://review.openstack.org/637009
17:15:01 openstackgerrit melanie witt proposed openstack/nova master: WIP Remove project_only=True from database queries https://review.openstack.org/637010
17:15:45 cdent efried: if you simply want to fix it, you can implement __deepcopy__ on the ProviderTree class
17:15:55 cdent but it's confusing that it hasn't come up before
17:18:04 openstackgerrit Merged openstack/nova master: Don't set bandwidth limits for vhostuser, hostdev interfaces https://review.openstack.org/635170
17:25:10 efried cdent: Exactly so.
17:27:26 efried Aside: I, for one, am looking forward to hearing all the gweilo trying to pronounce the U release name, whatever it winds up being :P
17:27:38 efried self included, of course
17:29:50 artom efried, heh, we'll just cop out and call it U
17:32:10 cfriesen of the "exception.DeviceNotFound" exception.
17:32:10 cfriesen melwitt: mriedem: question about https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L416-L422 I'm seeing this (on pike): http://paste.openstack.org/show/745113/ Which I think means that the original exception is lost (see last line in paste), and so the check for "if 'no target device' in errmsg" will never actually succeed. This results in me hitting the "libvirt.libvirtError" exception instead
17:33:43 melwitt looking
17:37:51 efried gibi: I don't want to delay this another day, but I still don't get it. Are you around to hold my hand and explain it?
17:41:29 melwitt cfriesen: the last line looks like it did the right thing, raised DeviceNotFound after catching 'no target device'. but the previous traceback is different, 'internal error' with a 'not found' message, which isn't handled in any way
17:42:14 efried gibi: nm, I think I see it now. And I am suitably shocked that this is the case.
17:42:25 cfriesen melwitt: it looks like the intent in LibvirtDriver.detach_volume() was to swallow the DeviceNotFound exception, but I seem to be hitting the libvirtError exception path which gets re-raised and handled in DriverVolumeBlockDevice.driver_detach
17:44:01 artom Weird question, but does oslo.config support *writing* config files? Sorta like python's configparser, but hopefully without getting rid of comments when writing (which configparser does :( )
17:44:02 melwitt cfriesen: right. it looks like you're hitting an unexpected combination where you have an libvirt 'internal error' with a 'not found' message, whereas we expect a 'operation failed' libvirt error to go with the 'not found' message
17:45:24 cfriesen got it...I hadn't clued in we were dealing with two separate exceptions
17:46:19 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: Use the correct mdev allocated from the pGPU https://review.openstack.org/636591
17:46:20 openstackgerrit Sylvain Bauza proposed openstack/nova master: Add functional test for libvirt vgpu reshape https://review.openstack.org/631559
17:46:20 openstackgerrit Sylvain Bauza proposed openstack/nova master: libvirt: implement reshaper for vgpu https://review.openstack.org/599208
17:46:38 bauzas efried: updated based on our discussions ^
17:46:55 bauzas now the pGPU finding is a dependency for the reshape patch
17:47:21 bauzas because if not, it would mean we would need to trigger another reshape for allocations wrongly set
17:47:45 melwitt cfriesen: I expect the one you're getting is VIR_ERR_INTERNAL_ERROR and we're only handling VIR_ERR_OPERATION_FAILED and VIR_ERR_INVALID_ARG
17:48:41 cfriesen melwitt: thanks, that gives me something to dig into at least.
17:50:33 melwitt cfriesen: np. looks like we might need a 'if errcode in (libvirt.VIR_ERR_OPERATION_FAILED, libvirt.VIR_ERR_INTERNAL_ERROR):' on L409
17:50:52 cfriesen melwitt: it looks like we expect the detach from the persistent config to sometimes fail...what would cause that?
17:51:47 melwitt cfriesen: it's only expected if the device is not there (already detached). that is, libvirt will raise an error if it's already detached

Earlier   Later