Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-04
20:22:46 melwitt clarkb: I think python-pastedeploy isn't being installed from the uca repo based on this http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/devstacklog.txt.gz#_2017-10-04_01_49_38_030 and trying installing it locally I'm able to do "from paste import deploy". so AFAICT the package isn't broken
20:24:15 dansmith jaypipes: edleafe: slap me if I missed something on any of these comments: https://review.openstack.org/#/c/486215
20:26:57 openstackgerrit Merged openstack/nova master: [placement] Update the placement deployment instructions https://review.openstack.org/469048
20:27:28 openstackgerrit Merged openstack/nova master: api-ref: remove redundant preserve_ephemeral mention from rebuild docs https://review.openstack.org/509273
20:27:56 openstackgerrit Merged openstack/nova master: api-ref: add note about rebuild not replacing volume-backed root disk https://review.openstack.org/509282
20:29:46 mriedem aha, found a bug in a test
20:32:10 penick Take that, stupid test
20:32:21 mriedem yeah!
20:34:39 melwitt does anyone know anything about wsgi_scripts in setup.cfg? I found that it's a directive for pbr to install scripts and they get installed to what setuptools considers to be the "install directory" and they end up in /usr/local/bin
20:34:59 melwitt is there any reason installing them that way would make it so they can't import modules that had been installed via apt-get?
20:37:07 mriedem sounds like a sdague question
20:38:15 sdague melwitt: import path will be determined by the python interpretter
20:38:26 sdague what's the execute line on it?
20:39:45 openstackgerrit Matt Riedemann proposed openstack/nova master: Avoid redundant BDM lookup in check_can_live_migrate_source https://review.openstack.org/509633
20:39:47 edleafe dansmith: tempting, but I think those are good points
20:40:10 dansmith edleafe: \o/
20:40:30 melwitt sdague: on the script? I'm not sure, it's generated by pbr from this https://github.com/openstack/keystone/blob/d20a3e971f5665604a739166987ed64e12c8de3d/keystone/server/wsgi.py#L36
20:40:38 sdague right
20:40:47 sdague on my devstack it's #!/usr/bin/python
20:41:13 sdague the only thing I can imagine is if it got a weird value during build
20:41:18 melwitt I'm trying to debug a ceph job failure so I don't have info like that for that env
20:41:35 sdague melwitt: url?
20:41:44 melwitt the only difference I see between it and normal gate jobs is paste deploy is being installed from apt-get instead of pip
20:42:10 sdague that shouldn't really matter, as long as we aren't talking about a venv
20:42:13 melwitt sdague: here's how keystone fails to start http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/screen-keystone.txt.gz
20:43:13 jaypipes dansmith: no, I think those are accurate issues you've identified.
20:43:21 melwitt sdague: I don't understand how it's not able to import paste deploy if paste deploy is installed (with apt-get)
20:43:52 melwitt and I don't think it's double installed with apt-get AND pip. I think the pip installs are skipped because it was already installed with apt-get
20:44:31 melwitt like this is it being skipped later http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/devstacklog.txt.gz#_2017-10-04_01_50_59_811
20:44:36 sdague melwitt: so... the more concerning thing is paste gets deployed again
20:44:51 sdague I think this is the python namespace issue
20:45:08 sdague paste & paste-deploy (which is really paste.deploy using a python namespace)
20:45:14 sdague end up being installed by apt
20:45:23 sdague then pip install new paste
20:45:27 sdague in /usr/local/lib
20:45:41 sdague which now means the search path for paste.* is in /usr/local/lib, the new one
20:45:59 sdague paste-deploy is not installed again, because the version looks the same to pip
20:46:08 sdague but it's in a different namespaced part of the tree
20:46:16 sdague so when you start importing it's not visible
20:46:17 melwitt oh, I see. damn
20:46:26 sdague this is why we got oslo to stop doing namespaces
20:46:33 melwitt I was focused only on paste.deploy
20:46:36 sdague yeh
20:46:59 sdague when I saw the upgrade of paste, that reminded me of this funky issue
20:47:17 sdague for this reason we try really hard to install very little python from system packages
20:47:23 sdague because you end up in exactly this situation
20:47:36 melwitt okay, so paste install wasn't skipped by pip for already existing because the version probably didn't satisfy
20:47:37 sdague I think the answer is to remove paste & paste-deploy from the dpkg list
20:47:44 sdague yes
20:47:48 sdague because of upper-constraints
20:47:49 melwitt guuhh
20:47:51 melwitt okay
20:48:14 clarkb isn't paste-deploy coming in transitively in the dpkg list though? that could get tricky to remove
20:48:23 melwitt it's coming in through ceph package
20:48:29 sdague http://logs.openstack.org/32/508432/1/check/gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-xenial/575e932/logs/devstacklog.txt.gz#_2017-10-04_01_51_18_464
20:48:59 sdague melwitt: yeh, well that's going to be non compatible with the way we do things
20:49:36 sdague you could hack in a: pip install -U --force paste.deploy
20:49:41 sdague at the right part of the process
20:50:35 sdague that's probably the best option if the ceph package is dragging that stuff in
20:51:22 melwitt sdague: okay, yeah thanks. I'll see where I could put something like that in the devstack ceph plugin
20:51:35 melwitt thanks for your help cracking this
20:55:55 mriedem omfg cells meeting in 5 minutes
20:56:03 melwitt !!!
20:56:04 openstack melwitt: Error: "!!" is not a valid command.
20:56:13 melwitt haha
20:56:32 melwitt my alarm alarmed openstack too
21:03:14 openstackgerrit Eric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif https://review.openstack.org/422512
21:22:16 edleafe dansmith: I may get to slap you yet
21:23:13 edleafe dansmith: The test removed from test_filter_scheduler was testing pre-pike conductors that didn't pass instance_uuids. Now that we're in Queens, that test is no longer needed
21:24:15 cali_boxer hello all
21:29:59 mriedem edleafe: if the test is no longer needed, did you also remove the code that's checking for empty instance_uuids?
21:30:09 mriedem edleafe: and if so, that should be split into a separate change
21:32:37 dansmith edleafe: the rpc api still allows for it to be empty right?
21:32:47 melwitt hm, devstack post-config happens after keystone is started. would not have expected that
21:33:02 dansmith edleafe: you don't just get to drop rpc compat when it's been long enough, you have to do the version work
21:33:13 clarkb melwitt: we were just talking about why that is in the keystone channel
21:33:25 clarkb melwitt: long story short there is no default default domain
21:33:34 dansmith edleafe: otherwise people that have version mismatches get uncool error messages instead of "this is too old for that"
21:33:36 clarkb melwitt: so you have to get keystone fully up and running before you can configure anything else
21:33:51 dansmith edleafe: I just dealt with one of those yesterday, where they thought they had upgraded conductor but hadn't
21:34:14 melwitt clarkb: hah, good coincidence. thanks for sharing
21:36:28 mriedem rybridges: ^
21:36:28 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: note that project_id filter only works with all_tenants https://review.openstack.org/509650
21:42:38 openstackgerrit Merged openstack/nova master: Remove unused get_all_instance_*metadata methods https://review.openstack.org/508299
21:43:07 openstackgerrit Merged openstack/nova master: Remove old compat code from servers ViewBuilder._get_metadata https://review.openstack.org/508326
21:44:28 edleafe dansmith: so if filter_scheduler gets a request without instance_uiuds, it has to return the old-style host_states without alternates?
21:57:46 melwitt sigh, I need to be able to do something after devstack does pip installs but before keystone starts and I'm not seeing a way
21:57:53 melwitt (in a devstack plugin)
22:00:35 mriedem https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1196 ?
22:00:43 mriedem melwitt: look at the run phases in stack.sh
22:00:48 mriedem those are the hook points i think
22:01:44 melwitt yeah. but I noticed keystone starts before post-config, which it's not supposed to (or originally wasn't). clarkb explained about that a little bit ago
22:01:46 mriedem it looks like everything is started after post-config, EXCEPT keystone
22:01:51 mriedem yup
22:02:25 mriedem ah yup here https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1064
22:03:07 mriedem i'm guessing b/c the accounts are needed to configure the other services?
22:03:14 mriedem like configure nova to talk to cinder/glance/neutron
22:03:24 mriedem you have to have those things created in keystone first, which means starting keystone
22:03:36 melwitt he said it's because there's no default default domain
22:03:46 mriedem so besides a pre-start phase being added,

Earlier   Later