| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-09 | |||
| 14:30:43 | vladikr | sfinucan, thanks | |
| 14:31:27 | johnthetubaguy | vladikr: thanks, lgtm | |
| 14:33:01 | openstackgerrit | John Garbutt proposed openstack/nova master: Stop leaking ports on port_create timeout https://review.openstack.org/347183 | |
| 14:34:57 | johnthetubaguy | BobBall: I am just imagining lots of -1s from the XenServer CI again? | |
| 14:35:43 | EmilienM | cdent: do we managed to promote TripleO CI with new nova but somehow we're unable to promote Puppet CI (which are btw much simpler to debug than tripleo) | |
| 14:36:03 | EmilienM | cdent: s/do we/we/ | |
| 14:36:59 | EmilienM | cdent: http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/nova/nova-placement-api.txt.gz#_2017-02-09_12_29_24_213 | |
| 14:37:01 | cdent | EmilienM: hrmm. Any clues? | |
| 14:37:02 | jaypipes | johnthetubaguy, bauzas: easy +2 here: https://review.openstack.org/#/c/430399/2 | |
| 14:37:09 | sfinucan | Sorry for what I'm about to do, but it should be the last time I've to do it (all the "risky" stuff is at the end and can bitrot for a while) | |
| 14:37:15 | EmilienM | cdent: I don't think placement api service is ready, I don't see the POST | |
| 14:38:51 | EmilienM | cdent: we're running nova from rc1 | |
| 14:39:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'use_ipv6' https://review.openstack.org/430911 | |
| 14:39:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: netutils: Ignore 'use_ipv6' for network templates https://review.openstack.org/430910 | |
| 14:39:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'ipv6_backend' https://review.openstack.org/430716 | |
| 14:39:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Ignore 'use_ipv6' for port filters https://review.openstack.org/430909 | |
| 14:39:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate various 'dns_', 'dhcp_' options https://review.openstack.org/431018 | |
| 14:39:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Fix indentation in conf/netconf https://review.openstack.org/430717 | |
| 14:39:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate various 'dns_' options https://review.openstack.org/431017 | |
| 14:39:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'ebtables_' options https://review.openstack.org/431019 | |
| 14:39:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate remaining options in conf/network https://review.openstack.org/431022 | |
| 14:39:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'allow_same_net_traffic' https://review.openstack.org/430913 | |
| 14:39:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Ignore 'allow_same_net_traffic' for port filters https://review.openstack.org/430912 | |
| 14:39:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Final cleanups in conf/network https://review.openstack.org/431023 | |
| 14:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'fake_network' https://review.openstack.org/431020 | |
| 14:39:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate various 'iptables_' options https://review.openstack.org/430915 | |
| 14:39:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'metadata_host', 'metadata_port' https://review.openstack.org/430914 | |
| 14:39:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'firewall_driver' https://review.openstack.org/412390 | |
| 14:39:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'send_arp_for_ha', 'send_arp_for_ha_count' https://review.openstack.org/431021 | |
| 14:40:18 | cdent | EmilienM: Looks like a similar situation to what we were seeing before: n-cpu is not finding the placement endpoint, never records inventory for the compute-node. When n-scheduler starts talking to the placement-api that's the first time it is being talked to, but there's no data there | |
| 14:40:43 | EmilienM | cdent: what is weird is that it passed on the other scenario | |
| 14:40:48 | EmilienM | cdent: which means we're racy now | |
| 14:41:04 | EmilienM | cdent: all our scenarios deploy nova/placement the same way | |
| 14:41:15 | sfinucan | johnthetubaguy, sdague: I think the first 10 or so of those are noddy but helpful nonetheless. Would appreciate ye taking a glance and (hopefully) sending them on their way | |
| 14:41:17 | EmilienM | owalsh: ^ fyi | |
| 14:41:42 | EmilienM | cdent: i'm currently investigating puppet catalog and resources orchestration | |
| 14:42:12 | sfinucan | They should clean up and mark virtually all of the nova-network opts for future removal, while the latter ones untangle the repurposed nova-net opts | |
| 14:42:14 | cdent | EmilienM: it would be great if we could get a summary, with times, of when things happen, because trying to inspect multiple logs at once is proving rather difficult | |
| 14:42:37 | johnthetubaguy | cdent: I wondering if we only try fetching the service catalog once? would we log something for that? | |
| 14:43:12 | EmilienM | cdent: sure let me prepare that: etherpad.openstack.org/p/puppet-nova-trunk-debug | |
| 14:43:40 | cdent | johnthetubaguy: I don't know. This is part of why I've been saying that I think our understanding of keystoneauth1 is incomplete. I personally haven't had a chance to fix that for myself yet. | |
| 14:44:43 | johnthetubaguy | cdent: yeah, I have been digging in there recently for the service token stuff, I know I keep getting lost in there still | |
| 14:46:40 | EmilienM | cdent: I tried to grep useful things : http://paste.fedoraproject.org/551748/66515901/ | |
| 14:47:22 | EmilienM | cdent: if it's too noisy I can reduce the file | |
| 14:47:36 | EmilienM | cdent: so that file is the catalog of a job that failed | |
| 14:49:39 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955 | |
| 14:50:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964 | |
| 14:50:34 | EmilienM | cdent: I reduced the catalog: http://paste.fedoraproject.org/551757/86651825/ | |
| 14:50:39 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add functional tests for cold migration to same host https://review.openstack.org/414926 | |
| 14:50:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042 | |
| 14:51:26 | cdent | EmilienM: I'm not sure that providing a log of the puppet activity (that is what that is isn't it?0 really tells us what we need to know. We need to understand the interactions of the various services, in a timeline, which means extracting signal from the logs of the various services, unforutnately. What I mean is: | |
| 14:52:50 | EmilienM | cdent: what gives the catalog is timestamps of when service run | |
| 14:52:56 | cdent | at time X something is registered in the catalog, at time Y something starts, at time Z n-cpu tries to talk to placement for the first time, at time A the placement service records its first hit, at time B n-cpu asks for a service catalog | |
| 14:52:59 | EmilienM | cdent: for ex, you can see when we configure cells, etc | |
| 14:53:01 | cdent | when the services _run_ is not enough | |
| 14:53:27 | cdent | it's when the actions happen that we're trying to detangle, isn't it? I'm not entirely sure. | |
| 14:53:36 | EmilienM | right | |
| 14:57:37 | sdague | the service catalog is embedded in the keystone token | |
| 14:58:03 | mordred | keypairs are per-AZ? | |
| 14:58:39 | sdague | johnthetubaguy / cdent perhaps there is a placement service call before it's registered? | |
| 14:58:45 | EmilienM | cdent: damn, now we promoted tripleo CI, I also see jobs flapping with placement service | |
| 14:58:53 | EmilienM | cdent: it's working sometimes but failing very often | |
| 14:59:25 | nicolasbock | Hi, I am trying to setup the placement-api | |
| 14:59:35 | cdent | sdague: would it not "try again"? | |
| 14:59:53 | nicolasbock | according to http://docs.openstack.org/developer/nova/placement.html there should be a wsgi script somewhere in the package | |
| 14:59:57 | sdague | cdent: while the token is valid? | |
| 15:00:15 | sdague | I thought typically for service users that token is only rotated on expiration | |
| 15:00:16 | nicolasbock | but I can't find that script | |
| 15:00:28 | nicolasbock | could someone give me a pointer where I can find it? | |
| 15:00:29 | cdent | sdague: i don't know, see above about not knowing enough | |
| 15:00:59 | sdague | mordred: there is nothing I know about in the keypair data model that makes it per AZ | |
| 15:01:28 | johnthetubaguy | mordred: +1 what sdague said, its per region, AFAIK | |
| 15:01:43 | mordred | cool. that's what I thought | |
| 15:01:58 | mordred | I got this bug report https://github.com/ansible/ansible/issues/21191 and it seemed crazy to me - I'll dig more, thanks | |
| 15:02:04 | dansmith | yep, and stored with the instance now, so not per cell either | |
| 15:02:12 | EmilienM | nicolasbock: /usr/bin/nova-placement-api | |
| 15:02:23 | nicolasbock | EmilienM, ah, cool! | |
| 15:02:25 | cdent | nicolasbock: a file called nova-placement-api will be automatically created as part of the install process. that is the wsgi script. where it ends up depends on your python setup | |
| 15:02:33 | cdent | aw, EmilienM beat me to it | |
| 15:02:39 | nicolasbock | cdent, thanks :) | |
| 15:02:53 | sdague | mordred: http://developer.openstack.org/api-ref/compute/?expanded=create-or-import-keypair-detail#create-or-import-keypair | |
| 15:03:03 | nicolasbock | I was expecting something like app.wsgi as it is done in some other services | |
| 15:03:23 | sdague | mordred: I have no idea why they believe that setting per AZ is a thing | |
| 15:03:23 | EmilienM | dansmith: hey good morning - we promoted TripleO Ci with nova/trunk because jobs passed but it's racy, I see some failure now. I have debug on http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/nova/nova-placement-api.txt.gz | |
| 15:03:35 | cdent | nicolasbock: what's being done with placement is an artifact of pbr | |
| 15:03:59 | dansmith | EmilienM: yeah, I've been watching. I have no ideas at the moment | |
| 15:04:13 | EmilienM | dansmith: ok, I keep digging | |
| 15:04:17 | nicolasbock | cdent, I see. I | |
| 15:04:36 | nicolasbock | is there a timetable for when placement will be split off into a separate entity? | |
| 15:04:45 | cdent | dansmith, EmilienM: if sdague is right that a valid token will not try again to get a service catalog and that service catalog doesn't yet have a placement entry, then that would explain things? | |
| 15:05:03 | cdent | nicolasbock: no, not yet. It's on the agenda for ptg, but nothing solid or detailed yet | |
| 15:05:10 | mordred | sdague: what confuses me more is that they seem to be getting availability_zone back in their response dict | |
| 15:05:11 | sdague | mordred could probably answer that keystoneauth1 assumption best | |
| 15:05:13 | nicolasbock | cdent, ok | |
| 15:05:24 | sdague | mordred: from what provider? | |
| 15:05:39 | sdague | mordred: that's definitely no where near upstream code | |
| 15:06:00 | mordred | sdague: unknown. I'll be asking- and telling them I will not support az for that resource | |
| 15:06:08 | mordred | sdague: what keystoneauth assumption? | |
| 15:06:09 | EmilienM | cdent, sdague: http://logs.openstack.org/15/424915/13/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/42fa9b9/logs/keystone-resources.txt.gz | |
| 15:06:39 | cdent | EmilienM: but at what time is that list made? | |