Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-16
15:31:02 dansmith no
15:31:20 dansmith I think grenade didn't catch it because it installs a new nova into /opt/stack/new and doesn't have the migrate_repo overlay with the old one
15:31:46 dansmith even if it did, it would use git as its package manager and it would remove the old one during an update I think
15:32:19 mriedem gah
15:32:39 mriedem should fix that gap at some point but i don't have an easy solution off the top of my head
15:32:59 dansmith yeah, it's actually weird that the same thing didn't happen with their packages already, actually
15:33:20 sdague dansmith: did we ever actually have a backported migration?
15:33:42 sdague I know the facility has existed for a while, but it's not regularly used
15:33:44 dansmith sdague: well, that's what I was trying to think ... I feel like we must've had one at least, but I'm not positive
15:33:50 dansmith sdague: yeah, very infrequently if ever
15:34:23 openstack Launchpad bug 1665370 in tripleo "api db_sync fails on upgrade due to duplicate migration files" [Critical,Triaged] - Assigned to Steven Hardy (shardy)
15:34:23 sdague so, I'm trying to understand the bug - https://bugs.launchpad.net/nova/+bug/1665370
15:34:46 dansmith sdague: if you use python setup.py install,
15:34:56 dansmith sdague: you'll end up with two 21_ migrations, you get that right?
15:35:39 mdbooth bauzas: You around?
15:35:45 sdague dansmith: i'm not sure I do
15:35:57 sdague because won't the new rpm file remove the old one?
15:36:04 mriedem sdague: different names
15:36:13 shardy hey folks, thanks for looking into this - I'm around if you need any more info from my environment
15:36:22 mriedem 021_placeholder in ocata, 021_build_something_fix in newton
15:36:22 shardy and/or to test any fixes
15:36:24 dansmith sdague: not rpm, I said setup.py
15:36:35 sdague dansmith: ok, how do you get the filesystem
15:36:43 sdague maybe that's where I'm trying to understand
15:36:48 dansmith sdague: the confusing thing is that I think the rpm should be removing the old one even in their case
15:37:10 sdague I don't understand how you create a filesystem where they both are there
15:37:20 dansmith sdague: I'm just saying, take a system, setup.py install newton, then ocata, you'll end up with two 21_ files because you don't have a package manager in the mix
15:37:28 mdbooth bauzas: I'm trying to join some dots and I'm missing something. When ComputeManager calls update_instance_info(instance), I can see that the scheduler squirrels that instance away in host_state. However, I can't see anywhere that the information is actually used. I'm assuming that something somewhere uses it to keep resource usage up to date in between complete syncs, but I'm being blind.
15:37:36 mriedem we don't pip install nova during grenade?
15:37:41 mriedem i.e. pip install .
15:37:48 sdague mriedem: we do
15:37:49 dansmith mriedem: we "develop" install it don't we?
15:37:58 dansmith which is an egg link
15:38:02 mriedem to put the files in /var/lib/python27/site-packages/nova/..
15:38:16 sdague mriedem: oh, we do do dev install
15:38:20 bauzas mriedem: I think we pip develop
15:38:22 dansmith yeah so dev install won't catch this
15:38:23 sdague that might be isolating
15:38:26 dansmith right
15:38:30 mriedem shart
15:38:46 dansmith the thing is,
15:38:47 bauzas devstack too
15:38:50 dansmith they install from rpms,
15:38:59 dansmith so they should be removing the old 21 and installing the new one, AFAIK
15:38:59 bauzas mdbooth: not sure I get your problem
15:39:04 sdague dansmith: right
15:39:13 sdague dansmith: the thing I could imagine is stale pyc files
15:39:20 dansmith sdague: yeah, could be that
15:39:23 dansmith shardy: owalsh ^
15:39:38 owalsh dansmith: well, rpm should be clearing out .pyc/pyo files too
15:39:49 mdbooth bauzas: I'm assuming that when ComputeManager calls update_instance_info(instance), it somehow updates the host_state data which is ultimately used by DiskFilter
15:39:52 bauzas mdbooth: oh gotcha
15:39:56 mdbooth bauzas: Is that correct?
15:39:58 BobBall mriedem johnthetubaguy: I'd be quite happy to deprecate the vif driver config option - BUT - is it used by Rackspace with Quark?
15:40:02 dansmith owalsh: so maybe before we go tweaking stuff in tree we should try to figure out why the cleaning isn't happening?
15:40:05 bauzas mdbooth: yeah, some filters
15:40:08 owalsh dansmith: grabbing the bits to create a local rpm now...
15:40:14 mdbooth I'm just trying to follow the whole code path, because I suspect there's a bug somewhere in it
15:40:18 shardy dansmith, owalsh: actually this could be a packaging issue:
15:40:20 shardy http://paste.openstack.org/show/599261/
15:40:21 mdbooth But I can't even find the code path :)
15:40:21 bauzas mdbooth: not sure about the DiskFilter, but some others
15:40:34 bauzas mdbooth: sec then
15:40:35 shardy the new python-nova doesn't seem to have obsoleted the old one
15:40:36 sdague dansmith: yeh, because I can't imagine there is another way to fix this, those placement backports are important
15:40:39 BobBall mriedem: johnthetubaguy: Also very happy to change the default to the OVS driver
15:40:48 dansmith shardy: maybe building the rpm again in an unclean space or something?
15:40:54 sdague I guess the only thing you could do is not change 21_ to be the new name
15:40:59 sdague it just stays placehold
15:41:11 owalsh shardy: you mean both ocata & newton are installed?
15:41:22 bauzas mdbooth: found one at least
15:41:24 dansmith sdague: yeah, although I think when we made up this procedure we were supposed to put it in both places in master, which didn't happen here, but agree, it's probably not the root cause
15:41:32 shardy dansmith: not sure, these are dlrn packages, I can try locally rebuilding with dlrn to check tho
15:41:46 shardy owalsh: rpm -qa seems to indicate that, which is a surprise
15:41:55 sdague dansmith: ok, I didn't realize the double master policy
15:42:04 bauzas mdbooth: eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L89
15:42:08 dansmith sdague: we never do this though, so not surprising we didn't do it right
15:42:10 sdague shardy: yeh, that looks like something unclean about the build
15:42:22 sdague dansmith: yeh, agreed
15:42:32 sdague black swans are what they are
15:42:32 shardy sdague: yeah, I'll go and take a closer look at that, sorry for not spotting it sooner
15:42:40 mdbooth bauzas: Ah... I was expecting it to use the instance info to update usage totals
15:42:41 shardy I'll leave the bug open for now until I figure it out
15:42:44 mdbooth Does it not do that?
15:43:40 mdbooth Also, instances is accessed there as a property, so I may have been grepping the wrong thing
15:44:23 mriedem BobBall: i'd like johnthetubaguy's opinion on that too
15:44:45 mriedem but it seems the default vif driver should be the ovs one if use_neutron=True by default in ocata
15:48:02 mriedem sdague: is there a reason we can't pip install rather than pip develop in a grenade run?
15:48:03 mdbooth bauzas: The code in question is here: https://review.openstack.org/#/c/430091/4/nova/scheduler/filters/disk_filter.py
15:48:13 mdbooth It's examining host_state.total_usable_disk_gb
15:48:37 dansmith mriedem: so I'm not sure the dev install is all that unusual,
15:48:53 dansmith mriedem: because the people I know that deploy from git do so in venvs, which is kinda how this behaves
15:49:10 mriedem yeah i thought about venvs
15:49:18 mriedem that's what bluebox does
15:49:23 mdbooth AFAICT that value is only ever set by a periodic task, and not if ComputeManager is sending updates?
15:49:27 sdague mriedem: the whole reason for the dev install is so that you can actually hack on and fix things
15:49:35 mdbooth bauzas: TBH I'm pretty confused
15:49:36 sdague it's kind of the purpose of devstack
15:49:40 mriedem sdague: yeah but we wouldn't do that in a CI run
15:49:47 dansmith mriedem: then it's different from local

Earlier   Later