Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-16
15:38:16 sdague mriedem: oh, we do do dev install
15:38:20 bauzas mriedem: I think we pip develop
15:38:22 dansmith yeah so dev install won't catch this
15:38:23 sdague that might be isolating
15:38:26 dansmith right
15:38:30 mriedem shart
15:38:46 dansmith the thing is,
15:38:47 bauzas devstack too
15:38:50 dansmith they install from rpms,
15:38:59 dansmith so they should be removing the old 21 and installing the new one, AFAIK
15:38:59 bauzas mdbooth: not sure I get your problem
15:39:04 sdague dansmith: right
15:39:13 sdague dansmith: the thing I could imagine is stale pyc files
15:39:20 dansmith sdague: yeah, could be that
15:39:23 dansmith shardy: owalsh ^
15:39:38 owalsh dansmith: well, rpm should be clearing out .pyc/pyo files too
15:39:49 mdbooth bauzas: I'm assuming that when ComputeManager calls update_instance_info(instance), it somehow updates the host_state data which is ultimately used by DiskFilter
15:39:52 bauzas mdbooth: oh gotcha
15:39:56 mdbooth bauzas: Is that correct?
15:39:58 BobBall mriedem johnthetubaguy: I'd be quite happy to deprecate the vif driver config option - BUT - is it used by Rackspace with Quark?
15:40:02 dansmith owalsh: so maybe before we go tweaking stuff in tree we should try to figure out why the cleaning isn't happening?
15:40:05 bauzas mdbooth: yeah, some filters
15:40:08 owalsh dansmith: grabbing the bits to create a local rpm now...
15:40:14 mdbooth I'm just trying to follow the whole code path, because I suspect there's a bug somewhere in it
15:40:18 shardy dansmith, owalsh: actually this could be a packaging issue:
15:40:20 shardy http://paste.openstack.org/show/599261/
15:40:21 mdbooth But I can't even find the code path :)
15:40:21 bauzas mdbooth: not sure about the DiskFilter, but some others
15:40:34 bauzas mdbooth: sec then
15:40:35 shardy the new python-nova doesn't seem to have obsoleted the old one
15:40:36 sdague dansmith: yeh, because I can't imagine there is another way to fix this, those placement backports are important
15:40:39 BobBall mriedem: johnthetubaguy: Also very happy to change the default to the OVS driver
15:40:48 dansmith shardy: maybe building the rpm again in an unclean space or something?
15:40:54 sdague I guess the only thing you could do is not change 21_ to be the new name
15:40:59 sdague it just stays placehold
15:41:11 owalsh shardy: you mean both ocata & newton are installed?
15:41:22 bauzas mdbooth: found one at least
15:41:24 dansmith sdague: yeah, although I think when we made up this procedure we were supposed to put it in both places in master, which didn't happen here, but agree, it's probably not the root cause
15:41:32 shardy dansmith: not sure, these are dlrn packages, I can try locally rebuilding with dlrn to check tho
15:41:46 shardy owalsh: rpm -qa seems to indicate that, which is a surprise
15:41:55 sdague dansmith: ok, I didn't realize the double master policy
15:42:04 bauzas mdbooth: eg. https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L89
15:42:08 dansmith sdague: we never do this though, so not surprising we didn't do it right
15:42:10 sdague shardy: yeh, that looks like something unclean about the build
15:42:22 sdague dansmith: yeh, agreed
15:42:32 sdague black swans are what they are
15:42:32 shardy sdague: yeah, I'll go and take a closer look at that, sorry for not spotting it sooner
15:42:40 mdbooth bauzas: Ah... I was expecting it to use the instance info to update usage totals
15:42:41 shardy I'll leave the bug open for now until I figure it out
15:42:44 mdbooth Does it not do that?
15:43:40 mdbooth Also, instances is accessed there as a property, so I may have been grepping the wrong thing
15:44:23 mriedem BobBall: i'd like johnthetubaguy's opinion on that too
15:44:45 mriedem but it seems the default vif driver should be the ovs one if use_neutron=True by default in ocata
15:48:02 mriedem sdague: is there a reason we can't pip install rather than pip develop in a grenade run?
15:48:03 mdbooth bauzas: The code in question is here: https://review.openstack.org/#/c/430091/4/nova/scheduler/filters/disk_filter.py
15:48:13 mdbooth It's examining host_state.total_usable_disk_gb
15:48:37 dansmith mriedem: so I'm not sure the dev install is all that unusual,
15:48:53 dansmith mriedem: because the people I know that deploy from git do so in venvs, which is kinda how this behaves
15:49:10 mriedem yeah i thought about venvs
15:49:18 mriedem that's what bluebox does
15:49:23 mdbooth AFAICT that value is only ever set by a periodic task, and not if ComputeManager is sending updates?
15:49:27 sdague mriedem: the whole reason for the dev install is so that you can actually hack on and fix things
15:49:35 mdbooth bauzas: TBH I'm pretty confused
15:49:36 sdague it's kind of the purpose of devstack
15:49:40 mriedem sdague: yeah but we wouldn't do that in a CI run
15:49:47 dansmith mriedem: then it's different from local
15:49:53 mriedem seems d-g could tell us to actually install
15:49:56 sdague mriedem: sure, but then you end up with another difference between CI and local
15:50:07 sdague and people get all mad why reproduce is not a thing
15:50:11 dansmith yeah, which isn't great, especially if what we have is testing a legit mode
15:50:17 dansmith which I think it is because of venvs
15:50:23 mriedem fine fine
15:50:29 mriedem just spit balling
15:50:34 sdague mriedem: yeh, sure
15:50:44 mriedem i'll go back to trashing things in this roundtable
15:50:57 bauzas mdbooth: wait, which value ?
15:51:09 bauzas mdbooth: you're showing me something not related to the instance
15:51:14 sdague this is just one of those places where we have to remember that testing is a model of production, and as a model, we've made certain approximations
15:51:37 mdbooth bauzas: Right. I was trying to find an association between those 2 things, and I can't find one.
15:51:41 mriedem at the least i guess we need something doc'ed for this since it was missed
15:51:54 mriedem so in 2 years from now when we do another one of these we can consult the guide
15:51:57 mdbooth I was assuming it exists, because otherwise I don't see how that value is updated exception by a heavyweight periodic task.
15:52:09 bauzas mdbooth: so, the HostState is generated/updated at every request in the HostManager
15:52:12 mdbooth However, that's just assumption on my part.
15:52:25 bauzas and then we pass it to the filter for each compute
15:54:10 mdbooth bauzas: We're not calling back to the compute host for each request though, right?
15:54:42 mdbooth bauzas: The model is occasionally we get real values by inspecting the compute host, and in between we do a best effort ?
15:55:30 jaypipes I see that mordred has stepped in a pile of mascot poo this morning.
15:58:34 bauzas mdbooth: I need to go find my daughter
15:58:52 bauzas mdbooth: but let's discuss in 1 hour if I'm there
16:02:20 mriedem jaypipes: link me
16:06:03 frickler BobBall: could you take a look at https://review.openstack.org/434666 please? second (different) error after xenserver recheck, both look unrelated to my patch
16:08:36 BobBall frickler: We believe the ssh timeout issue is one we have a fix for (found the probable fix yesterday) - but we're currently doing more testing on it before merging to the CI. I've not seen that subnet failure before though.
16:09:13 BobBall There was a cirros change which made the CI very sad this morning, so I don't have enough statistics after that was fixed to say whether there is another wider issue or bad luck.
16:09:47 BobBall frickler: Hit recheck again - we are monitoring things and when we have a few hours more worth of data hopefully we can say whether there is a new issue or not
16:16:53 mordred jaypipes: could soemeone please revoke my email access?
16:16:57 mriedem bauzas: https://review.openstack.org/#/c/434552/
16:16:59 mriedem for rc2
16:17:42 mdbooth mordred jaypipes: Did I miss the juicy bits? Seemed civil to me.
16:18:53 mriedem nova got lumped in there i saw

Earlier   Later