| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-sdks - 2019-08-13 | |||
| 14:00:46 | efried | or | |
| 14:01:52 | efried | I guess it could be timing. If the code path that hits .nodes() happens much earlier than get_node(), and the service is still down, and we try to make the proxy and fail discovery... but we should try again a bit later, shouldn't we?? | |
| 14:02:14 | dtantsur | I guess we do? I'd fix the get_node problem first, then see where we stand | |
| 14:03:21 | efried | dtantsur: Yeah, see, the next patch in the series *should* be hitting that same get_node error, but it's not getting that far: https://logs.opendev.org/27/656027/21/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/60265c4/controller/logs/screen-n-cpu.txt.gz?level=ERROR | |
| 14:04:05 | dtantsur | it cannot connect to keystone, maybe that's the cause? | |
| 14:04:09 | efried | I suspect this just means the nova side needs to be more tolerant of service-not-yet-up. | |
| 14:04:15 | dtantsur | yep | |
| 14:08:51 | openstackgerrit | Merged openstack/openstacksdk master: DRY test_connection https://review.opendev.org/676009 | |
| 14:48:35 | efried | dustinc: ^ | |
| 14:52:16 | dustinc | efried, dtantsur: thanks, will catch up when I get to work in about an hour | |
| 14:57:14 | dtantsur | mordred, efried, https://review.opendev.org/#/c/676154/ passed now, please check | |
| 15:14:40 | openstackgerrit | Merged openstack/openstacksdk master: Fix discovery cache sharing https://review.opendev.org/675121 | |
| 15:35:31 | efried | dtantsur: +A, thanks for the quick fix. | |
| 15:36:28 | dtantsur | yvw | |
| 15:47:55 | efried | dtantsur, mordred: still trying to nail down the difference in the discovery path between ksa and sdk | |
| 15:48:08 | efried | specifically here https://logs.opendev.org/27/656027/21/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/60265c4/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Aug_13_04_59_18_053245 | |
| 15:48:38 | efried | In this case we want ServiceUnavailable to be an exception, but I *think* it's proceeding and still trying to return a real proxy. | |
| 15:49:04 | efried | I think allow_version_hack=False would get us DiscoveryFailure here | |
| 15:49:41 | efried | ...but I'm not sure whether that would help :( | |
| 15:55:04 | efried | mordred: I... kind of suspect we should be defaulting allow_version_hack=False from the sdk side. AFAICT it's always True today - even for the code paths where we're not explicitly setting it. | |
| 16:23:00 | efried | dustinc: When you're about, we should talk. | |
| 16:29:14 | dustinc | efried: around now | |
| 16:29:55 | efried | dustinc: The nova/sdk series needs to be passing the ironic CI job before it's going to fly | |
| 16:30:25 | efried | before yesterday, with the fields thing, the bottom patch was passing, but the second one (and up) was failing. | |
| 16:30:35 | efried | and I still haven't been able to figure out exactly why. | |
| 16:34:31 | dustinc | Thanks, I have not been giving it a lot of attention recently but am working on it today | |
| 16:35:40 | efried | dustinc: In the CI env, the n-cpu service comes up before ironic | |
| 16:35:47 | efried | so the first few attempts to connect will fail. | |
| 16:35:57 | efried | This is a "normal" run: https://logs.opendev.org/40/675940/1/check/ironicclient-tempest/f57c0f3/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Aug_12_16_38_40_790400 | |
| 16:37:07 | efried | And that's what things would look like, more or less, on the bottom patch (before the fields thing) | |
| 16:37:24 | dustinc | thanks, checking it out right now | |
| 16:37:36 | efried | But then with the second patch, we would start seeing stuff like this https://logs.opendev.org/27/656027/21/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/60265c4/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Aug_13_04_59_18_053245 | |
| 16:37:37 | dustinc | (I knew about the fields support thing but totally forgot and thus added it back..) | |
| 16:38:12 | efried | My current suspicion is that it's because we're using different retries/timeout than before. | |
| 16:38:21 | dustinc | Hmm | |
| 16:46:25 | efried | dtantsur|afk, dustinc, mordred: I think now that we have retry opts in ksa config [1][2], we should deprecate nova's equivalents [3] via alias so that they're used for both the old (iroincclient) and new (sdk) code paths, which will assure us parity at least in that regard. | |
| 16:46:25 | efried | [1] https://review.opendev.org/666287 | |
| 16:46:25 | efried | [2] https://review.opendev.org/672930 | |
| 16:46:25 | efried | [3] https://opendev.org/openstack/nova/src/branch/master/nova/conf/ironic.py#L39-L63 | |
| 16:59:53 | openstackgerrit | Merged openstack/openstacksdk master: Add support for fields in baremetal get_* resources https://review.opendev.org/676154 | |
| 17:00:11 | openstackgerrit | Eric Fried proposed openstack/openstacksdk master: Bump keystoneauth1 minimum to 3.16.0 https://review.opendev.org/676247 | |
| 17:00:36 | efried | mordred, Shrews: I'm going to propose a sdk release now, mkay? | |
| 17:00:45 | efried | accumulated a bunch of fixes nova (at least) needs. | |
| 17:01:11 | mordred | efried: ++ | |
| 17:01:15 | Shrews | up to mordred, but i have no problem with it | |
| 17:01:43 | mordred | efried: actually ... | |
| 17:01:48 | Shrews | will be able to restart nodepool with the object fixes, which will be nice | |
| 17:01:55 | mordred | efried: should we also land https://review.opendev.org/#/c/674807/ too? | |
| 17:02:12 | mordred | (that also contains the keystoneauth dep bump) | |
| 17:03:07 | efried | mordred: could do, yes. Is there a reason that wasn't +W'd? | |
| 17:03:24 | efried | mordred: also, that one bumps to 3.15.0, but we need .16. to pick up both retry changes. | |
| 17:03:34 | efried | I could rebase my bump on that one and +W it. | |
| 17:03:47 | mordred | ++ | |
| 17:11:01 | openstackgerrit | Eric Fried proposed openstack/openstacksdk master: Bump keystoneauth1 minimum to 3.16.0 https://review.opendev.org/676247 | |
| 17:11:04 | efried | mordred: ^ | |
| 17:11:12 | efried | Shrews: too | |
| 17:15:33 | efried | dustinc: I'm going to start hacking at that thing I said earlier, about deprecating the existing ironic retry opts. There are other ways to get the right timeouts into the sdk proxy, but they're messier. | |
| 17:15:44 | efried | ...unless you want it | |
| 17:17:37 | dustinc | efried: go for it, I am about to push a small fix to the bottom patch | |
| 17:18:16 | efried | dustinc: are you taking out 'fields' for now? | |
| 17:18:39 | dustinc | I was going to leave it expecting the SDK release to add support | |
| 17:18:52 | dustinc | I can take it out for now if you want.. | |
| 17:19:40 | efried | dustinc: meh | |
| 17:19:50 | efried | I think we're not going to be able to land anything until we've got some of these issues sorted out | |
| 17:19:54 | efried | so probably no hurry | |
| 17:20:05 | efried | but you're going to want to put your series on top of this thing I'm doing | |
| 17:20:11 | efried | which will also require the new sdk release | |
| 17:20:21 | efried | for which I'm going to wait for the ksa bump | |
| 17:20:31 | efried | so we got a little while until all those pieces come together. | |
| 18:35:22 | openstackgerrit | Merged openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807 | |
| 20:35:43 | openstackgerrit | Merged openstack/openstacksdk master: Bump keystoneauth1 minimum to 3.16.0 https://review.opendev.org/676247 | |
| 20:44:33 | efried | openstacksdk release 0.34.0 proposed: https://review.opendev.org/676276 | |
| 22:50:30 | openstackgerrit | Merged openstack/openstacksdk master: Minor refactor cleanup of _make_proxy https://review.opendev.org/676015 | |
| 23:51:47 | openstackgerrit | Adrian Turjak proposed openstack/keystoneauth master: add support for auth_receipts and multi-method auth https://review.opendev.org/675049 | |
| #openstack-sdks - 2019-08-14 | |||
| 14:52:30 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: DNM Test patch for sphinx job https://review.opendev.org/676432 | |
| 15:04:24 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: DNM Test patch for sphinx job https://review.opendev.org/676432 | |
| 15:06:47 | openstackgerrit | Shubham Potale proposed openstack/openstacksdk master: Add support for nfv-orchestration service https://review.opendev.org/675604 | |
| 16:04:59 | Shrews | https://docs.openstack.org/swift/latest/api/object-expiration.html | |
| 16:05:06 | Shrews | that would have been helpful a few days ago | |
| 16:05:15 | Shrews | but yeah for existing | |
| 16:05:38 | Shrews | s/yeah/yay/ | |
| 16:07:51 | mordred | Shrews: you don't have all of the API docs memorized? | |
| 19:01:11 | openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Make block-storage work with proxies https://review.opendev.org/675382 | |
| 19:05:09 | efried | mordred: I think I need to pre-emptively and backward-compatibly fixture nova for stuff that changed in _make_proxy between 0.33.0 and 0.34.0 https://review.opendev.org/#/c/676457/1 | |
| 19:10:38 | mordred | efried: poop. oh - because of the mocking? | |
| 19:11:13 | mordred | (sorry - the sharable fixture is taking eversoslightly longer than I'd hoped - I may have uncovered a few cans with many worms) | |
| 19:12:39 | efried | mordred: I haven't fully nailed it down yet, just have a local repro. It's because of this code path: | |
| 19:12:39 | efried | # We don't know anything about this service, so the user is | |
| 19:12:39 | efried | # explicitly just using us for a passthrough REST adapter. | |
| 19:12:39 | efried | # Skip all the lower logic. | |
| 19:12:39 | efried | if not self.supported_versions: | |
| 19:12:40 | efried | temp_client = config.get_session_client( | |
| 19:12:40 | efried | self.service_type, | |
| 19:12:41 | efried | allow_version_hack=True, | |
| 19:12:41 | efried | ) | |
| 19:12:42 | efried | # trigger EndpointNotFound exception if this is bogus | |
| 19:12:42 | efried | temp_client.get_endpoint() | |
| 19:12:43 | efried | return temp_client | |
| 19:13:15 | efried | That temp_client.get_endpoint is raising MissingAuthPlugin from ksa | |
| 19:14:54 | efried | I have a fix... | |