Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-16
15:36:13 shardy hey folks, thanks for looking into this - I'm around if you need any more info from my environment
15:36:22 mriedem 021_placeholder in ocata, 021_build_something_fix in newton
15:36:22 shardy and/or to test any fixes
15:36:24 dansmith sdague: not rpm, I said setup.py
15:36:35 sdague dansmith: ok, how do you get the filesystem
15:36:43 sdague maybe that's where I'm trying to understand
15:36:48 dansmith sdague: the confusing thing is that I think the rpm should be removing the old one even in their case
15:37:10 sdague I don't understand how you create a filesystem where they both are there
15:37:20 dansmith sdague: I'm just saying, take a system, setup.py install newton, then ocata, you'll end up with two 21_ files because you don't have a package manager in the mix
15:37:28 mdbooth bauzas: I'm trying to join some dots and I'm missing something. When ComputeManager calls update_instance_info(instance), I can see that the scheduler squirrels that instance away in host_state. However, I can't see anywhere that the information is actually used. I'm assuming that something somewhere uses it to keep resource usage up to date in between complete syncs, but I'm being blind.
15:37:36 mriedem we don't pip install nova during grenade?
15:37:41 mriedem i.e. pip install .
15:37:48 sdague mriedem: we do
15:37:49 dansmith mriedem: we "develop" install it don't we?
15:37:58 dansmith which is an egg link
15:38:02 mriedem to put the files in /var/lib/python27/site-packages/nova/..
15:38:16 sdague mriedem: oh, we do do dev install
15:38:20 bauzas mriedem: I think we pip develop
15:38:22 dansmith yeah so dev install won't catch this
15:38:23 sdague that might be isolating
15:38:26 dansmith right
15:38:30 mriedem shart
15:38:46 dansmith the thing is,
15:38:47 bauzas devstack too
15:38:50 dansmith they install from rpms,
15:38:59 dansmith so they should be removing the old 21 and installing the new one, AFAIK
15:38:59 bauzas mdbooth: not sure I get your problem
15:39:04 sdague dansmith: right
15:39:13 sdague dansmith: the thing I could imagine is stale pyc files
15:39:20 dansmith sdague: yeah, could be that
15:39:23 dansmith shardy: owalsh ^
15:39:38 owalsh dansmith: well, rpm should be clearing out .pyc/pyo files too
15:39:49 mdbooth bauzas: I'm assuming that when ComputeManager calls update_instance_info(instance), it somehow updates the host_state data which is ultimately used by DiskFilter
15:39:52 bauzas mdbooth: oh gotcha
15:39:56 mdbooth bauzas: Is that correct?
15:39:58 BobBall mriedem johnthetubaguy: I'd be quite happy to deprecate the vif driver config option - BUT - is it used by Rackspace with Quark?
15:40:02 dansmith owalsh: so maybe before we go tweaking stuff in tree we should try to figure out why the cleaning isn't happening?
15:40:05 bauzas mdbooth: yeah, some filters
15:40:08 owalsh dansmith: grabbing the bits to create a local rpm now...
15:40:14 mdbooth I'm just trying to follow the whole code path, because I suspect there's a bug somewhere in it
15:40:18 shardy dansmith, owalsh: actually this could be a packaging issue:
15:40:20 shardy http://paste.openstack.org/show/599261/
15:40:21 mdbooth But I can't even find the code path :)
15:40:21 bauzas mdbooth: not sure about the DiskFilter, but some others
15:40:34 bauzas mdbooth: sec then
15:40:35 shardy the new python-nova doesn't seem to have obsoleted the old one
15:40:36 sdague dansmith: yeh, because I can't imagine there is another way to fix this, those placement backports are important
15:40:39 BobBall mriedem: johnthetubaguy: Also very happy to change the default to the OVS driver
15:40:48 dansmith shardy: maybe building the rpm again in an unclean space or something?
15:40:54 sdague I guess the only thing you could do is not change 21_ to be the new name
15:40:59 sdague it just stays placehold
15:41:11 owalsh shardy: you mean both ocata & newton are installed?
15:41:22 bauzas mdbooth: found one at least
15:41:24 dansmith sdague: yeah, although I think when we made up this procedure we were supposed to put it in both places in master, which didn't happen here, but agree, it's probably not the root cause
15:41:32 shardy dansmith: not sure, these are dlrn packages, I can try locally rebuilding with dlrn to check tho
15:41:46 shardy owalsh: rpm -qa seems to indicate that, which is a surprise
15:41:55 sdague dansmith: ok, I didn't realize the double master policy
15:42:04 bauzas mdbooth: eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L89
15:42:08 dansmith sdague: we never do this though, so not surprising we didn't do it right
15:42:10 sdague shardy: yeh, that looks like something unclean about the build
15:42:22 sdague dansmith: yeh, agreed
15:42:32 sdague black swans are what they are
15:42:32 shardy sdague: yeah, I'll go and take a closer look at that, sorry for not spotting it sooner
15:42:40 mdbooth bauzas: Ah... I was expecting it to use the instance info to update usage totals
15:42:41 shardy I'll leave the bug open for now until I figure it out
15:42:44 mdbooth Does it not do that?
15:43:40 mdbooth Also, instances is accessed there as a property, so I may have been grepping the wrong thing
15:44:23 mriedem BobBall: i'd like johnthetubaguy's opinion on that too
15:44:45 mriedem but it seems the default vif driver should be the ovs one if use_neutron=True by default in ocata
15:48:02 mriedem sdague: is there a reason we can't pip install rather than pip develop in a grenade run?
15:48:03 mdbooth bauzas: The code in question is here: https://review.openstack.org/#/c/430091/4/nova/scheduler/filters/disk_filter.py
15:48:13 mdbooth It's examining host_state.total_usable_disk_gb
15:48:37 dansmith mriedem: so I'm not sure the dev install is all that unusual,
15:48:53 dansmith mriedem: because the people I know that deploy from git do so in venvs, which is kinda how this behaves
15:49:10 mriedem yeah i thought about venvs
15:49:18 mriedem that's what bluebox does
15:49:23 mdbooth AFAICT that value is only ever set by a periodic task, and not if ComputeManager is sending updates?
15:49:27 sdague mriedem: the whole reason for the dev install is so that you can actually hack on and fix things
15:49:35 mdbooth bauzas: TBH I'm pretty confused
15:49:36 sdague it's kind of the purpose of devstack
15:49:40 mriedem sdague: yeah but we wouldn't do that in a CI run
15:49:47 dansmith mriedem: then it's different from local
15:49:53 mriedem seems d-g could tell us to actually install
15:49:56 sdague mriedem: sure, but then you end up with another difference between CI and local
15:50:07 sdague and people get all mad why reproduce is not a thing
15:50:11 dansmith yeah, which isn't great, especially if what we have is testing a legit mode
15:50:17 dansmith which I think it is because of venvs
15:50:23 mriedem fine fine
15:50:29 mriedem just spit balling
15:50:34 sdague mriedem: yeh, sure
15:50:44 mriedem i'll go back to trashing things in this roundtable
15:50:57 bauzas mdbooth: wait, which value ?
15:51:09 bauzas mdbooth: you're showing me something not related to the instance
15:51:14 sdague this is just one of those places where we have to remember that testing is a model of production, and as a model, we've made certain approximations
15:51:37 mdbooth bauzas: Right. I was trying to find an association between those 2 things, and I can't find one.
15:51:41 mriedem at the least i guess we need something doc'ed for this since it was missed
15:51:54 mriedem so in 2 years from now when we do another one of these we can consult the guide
15:51:57 mdbooth I was assuming it exists, because otherwise I don't see how that value is updated exception by a heavyweight periodic task.
15:52:09 bauzas mdbooth: so, the HostState is generated/updated at every request in the HostManager
15:52:12 mdbooth However, that's just assumption on my part.

Earlier   Later