| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-20 | |||
| 18:40:41 | mriedem | maybe i was the only one that heard my swearing | |
| 18:41:08 | cdent | edleafe: you can to check that the cn1uuid is in there, which in there? | |
| 18:41:13 | smcginnis | Oh yeah, I remember now. :) | |
| 18:41:14 | cdent | s/can/want/? | |
| 18:42:33 | edleafe | cdent: dunno, just thought that checking that it's in there wasn't definitive enough. Getting allocation_requests[0].keys()[0] or something like that | |
| 18:42:35 | melwitt | I'm able to repro the EndpointNotFound problem in an old devstack I have, and what I get from ksa_adapter.get_endpoint() in both the non-list and interface list cases is: "*** EmptyCatalog: The service catalog is empty." | |
| 18:43:08 | melwitt | even though I have glance in 'openstack endpoint list' | |
| 18:44:30 | cdent | edleafe: if you know the uuid in advance, it's generally easier to check for some key down its path. you can do environ and response expansions within the left hand side of a json path thing | |
| 18:45:04 | efried | melwitt: Can you show me your openstack endpoint list for glance? | |
| 18:45:11 | efried | or 'show' would be better | |
| 18:46:43 | efried | melwitt: The ksa_adapter is going to be narrowed down based on (possibly defaulted) config options. So as Matt was saying earlier, if for example your catalog has the admin endpoint, but your conf has (possibly by defaulting) internal and public, you'll get that EmptyCatalog/EndpointNotFound when you ask the adapter for an endpoint. | |
| 18:47:08 | cdent | edleafe, sorry, I'm doing the usual too many things at once and then writing two thoughts down in one sentence | |
| 18:47:25 | melwitt | efried: okay, sec | |
| 18:47:34 | cdent | edleafe: first example after the heading: https://gabbi.readthedocs.io/en/latest/jsonpath.html#substitution | |
| 18:47:50 | cdent | (well, only example) | |
| 18:49:20 | edleafe | cdent: So given the example paste, what would I put in the left side? IOW, what is "nested.structure"? | |
| 18:50:02 | melwitt | efried: http://paste.openstack.org/show/706465/ | |
| 18:50:38 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094 | |
| 18:50:42 | efried | melwitt: And the glance section of the conf? | |
| 18:51:04 | melwitt | no [glance] section, all default | |
| 18:51:20 | efried | melwitt: ...and the version document from the endpoint? (curl http://127.0.0.1/image) | |
| 18:51:57 | melwitt | http://paste.openstack.org/show/706468/ | |
| 18:52:23 | cdent | edleafe: $.allocation_requests[0].allocations.["$ENVIRON['CN1']"].resources.DISK_GB: 100 | |
| 18:52:29 | cdent | I think jay did this somewhere, will find example | |
| 18:53:26 | cdent | edleafe: $.allocation_requests..allocations["$ENVIRON['SS_UUID']"].resources[DISK_GB]: [100, 100] in the allocation-candidates.yaml | |
| 18:53:37 | efried | melwitt: Yup, that ought to work. | |
| 18:54:50 | efried | melwitt: You said you were using and old stack? What version of ksa, and what commit of nova? | |
| 18:55:33 | melwitt | stepping through the code in pdb, when I get to keystoneauth1/access/service_catalog.py(362)endpoint_data_for() the self._catalog is an empty list [], even for the 'public' interface | |
| 18:55:46 | melwitt | that's where it raises exceptions.EmptyCatalog('The service catalog is empty.') | |
| 18:55:49 | melwitt | sec | |
| 18:55:52 | edleafe | cdent: ok, thanks - that helps. | |
| 18:56:17 | melwitt | keystoneauth1==3.4.0 | |
| 18:56:45 | melwitt | commit a5a569d6670c29f995b1e8a2a2013471d57469d7 of nova | |
| 18:57:30 | melwitt | it raises exceptions.EmptyCatalog('The service catalog is empty.') for each of 'internal' and 'public' | |
| 18:57:34 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990 | |
| 18:57:35 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282 | |
| 18:57:35 | openstackgerrit | Dan Smith proposed openstack/nova master: Add require_tenant_aggregate request filter https://review.openstack.org/545002 | |
| 18:57:56 | dansmith | edleafe: your stuff in action: https://review.openstack.org/#/c/545002/12/nova/tests/functional/test_aggregates.py | |
| 18:58:15 | dansmith | L224 specifically | |
| 18:58:38 | efried | melwitt: Is that ocata? | |
| 18:59:31 | melwitt | no? | |
| 19:00:05 | efried | sorry, trying the wrong way to find where that is chronologically :) | |
| 19:00:11 | melwitt | master from Feb 12, the commit I linked is dated Feb 12 | |
| 19:00:41 | melwitt | it might have probably merged later | |
| 19:00:43 | efried | melwitt: Is this code path using a RequestContext? | |
| 19:00:47 | melwitt | but it's from around that time | |
| 19:01:00 | melwitt | yes, it's a RequestContext | |
| 19:01:10 | efried | What's in the RequestContext.service_catalog? | |
| 19:03:01 | melwitt | it's empty ... though I think that's my fault, I created an admin context to call glance.api_servers | |
| 19:05:35 | efried | Well, unless I'm mistaken, the get_endpoint() stuff winds up in whatever context you're using. So that would splain why it's empty in your pdb. | |
| 19:06:31 | melwitt | is the RequestContext.service_catalog supposed to get populated as a result of the get_endpoint() call? or is it supposed to be pre-populated before the get_endpoint call? | |
| 19:06:47 | efried | The latter | |
| 19:07:01 | melwitt | okay, yeah, then I've messed up this attempt to repro the problem | |
| 19:07:02 | efried | get_endpoint gets its information *from* the context. | |
| 19:08:26 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Parse placement forbidden traits query string https://review.openstack.org/554665 | |
| 19:08:53 | openstackgerrit | Giridhar Jayavelu proposed openstack/nova-specs master: VMware: place instances on resource pool https://review.openstack.org/549067 | |
| 19:13:21 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: stop lying to the RT when ironic is down https://review.openstack.org/545479 | |
| 19:13:23 | jroll | whee, that should be good now | |
| 19:13:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | |
| 19:13:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Preserve multiattach flag when refreshing connection_info https://review.openstack.org/554667 | |
| 19:16:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | |
| 19:16:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Preserve multiattach flag when refreshing connection_info https://review.openstack.org/554667 | |
| 19:24:14 | melwitt | okay, repro'd again by actually creating an instance, doing a virsh shutdown, then waiting for nova-compute to stop the instance, same error and again RequestContext.service_catalog = [] (I logged it from nova/image/glance.py) | |
| 19:25:04 | melwitt | so somewhere along the way, the context we pass isn't one with a properly populated service_catalog | |
| 19:31:41 | melwitt | uh oh, I think I know why. this path is being run through a periodic task which has been given an anonymous get_admin_context(), and when we eventually try to do something with a service_catalog in the RequestContext, there isn't going to be one | |
| 19:32:18 | efried | melwitt: That'd do it. Lemme dig up the patch that did the context-y stuff... | |
| 19:32:52 | efried | melwitt: https://review.openstack.org/#/c/490057/ | |
| 19:32:53 | melwitt | I'm not sure what the answer is here, other than gibi looking into whether we can avoid relying on service catalog stuff for sending notifications | |
| 19:33:19 | efried | melwitt: That patch ought to help us isolate which context is giving us grief here. | |
| 19:33:23 | mriedem | nova doesn't have admin creds to glance like we do for cinder and neutron, so you can't rely on that in a periodic either | |
| 19:33:25 | melwitt | because the context we use in periodic tasks is going to be a mostly empty admin one intended to read the database, etc. it's not going to have service catalog info in it | |
| 19:33:43 | melwitt | oh, okay. so we are ok for cinder and neutron then | |
| 19:33:54 | mriedem | if properly configured | |
| 19:34:06 | mriedem | you have to configure nova to talk to neutron with an admin role token for port binding, | |
| 19:34:18 | efried | Mm, yeah, it's coming back to me. I don't remember where else we've seen this, but the answer was: if you want this to work, you have to supply creds in the conf so that we can build a proper admin context. | |
| 19:34:19 | mriedem | the cinder <> admin config was added in queens for a related bug with periodic tasks | |
| 19:34:21 | mriedem | doing things with volumes | |
| 19:34:28 | efried | Yeah, what mriedem said. | |
| 19:34:33 | melwitt | gotcha, okay | |
| 19:34:44 | efried | And... is that okay? | |
| 19:34:51 | efried | or do we need to "fix" it? | |
| 19:35:10 | mriedem | for notifications, we really shouldn't have to hit a REST API every time we send a notification, because that's kind of crazy | |
| 19:35:22 | openstackgerrit | sean mooney proposed openstack/nova master: add mtu to libvirt xml for ethernet and bridge types https://review.openstack.org/553072 | |
| 19:35:39 | efried | Yeah, I'll agree with that. Probably the first thing to look into, then - why we need to talk to glance to send a notification. | |
| 19:35:54 | mriedem | i've mentioned this before, | |
| 19:35:55 | melwitt | yeah, gibi said he's going to investigate that tomorrrow | |
| 19:36:08 | mriedem | but whenever we construct a glance "client" object in-tree, it goes thorugh the 'get endpoint url' stuff | |
| 19:36:35 | mriedem | notifications goes through info_from_instance to build a payload, | |
| 19:36:41 | mriedem | which gets an image ref URL | |
| 19:36:47 | efried | mriedem: Yeah, I remember you mentioning it was doing it like 4000 times in a devstack run, or something. | |
| 19:36:52 | mriedem | efried: yup | |
| 19:36:53 | efried | s/devstack/tempest/ | |
| 19:36:58 | mriedem | we f'ing love to hit glance | |
| 19:37:24 | mriedem | if [glance]/api_servers is set, we just build a static string based on that | |
| 19:37:29 | sean-k-mooney[m] | mriedem: so the api call to glance when sending the notificaiton is from creating the glance client object? | |
| 19:37:33 | mriedem | but if it's not set, we go through the ksa magik | |
| 19:37:45 | efried | Which *should* be using cached values, I thought. | |
| 19:37:55 | mriedem | maybe it is | |
| 19:37:55 | efried | i.e. we're not actually hitting the API 4000 times. | |
| 19:38:02 | mriedem | that would be nice to know | |