| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-23 | |||
| 16:46:59 | mriedem | which cell is c244e139-98fe-44eb-a971-c067954da026 in ? | |
| 16:47:03 | mriedem | cell1 or cell0? | |
| 16:47:16 | KeithMnemonic | for not deleted it works as expected. let me check one sec | |
| 16:49:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | |
| 16:51:35 | KeithMnemonic | cell1 it seems | |
| 16:52:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | |
| 16:52:52 | mriedem | hmm, i wonder if there is something going on with the multi-cell instance listing stuff that has some marker logic in it | |
| 16:53:08 | KeithMnemonic | let me try it with a market in cell0 | |
| 16:53:13 | KeithMnemonic | i have one instance there | |
| 16:53:16 | melwitt | mriedem: uh oh https://github.com/openstack/nova/blob/master/nova/compute/instance_list.py#L59 | |
| 16:53:34 | melwitt | oh nevermind | |
| 16:53:36 | mriedem | was just looking at that code, | |
| 16:53:41 | mriedem | but yeah a deleted instance will still have a mapping | |
| 16:53:43 | melwitt | there should still be an instance mapping? or no | |
| 16:53:43 | mriedem | until archived | |
| 16:53:45 | melwitt | yeah | |
| 16:53:52 | mriedem | and https://github.com/openstack/nova/blob/master/nova/compute/instance_list.py#L63 | |
| 16:53:59 | mriedem | matches the db api to read deleted markers | |
| 16:54:05 | melwitt | yeah hm | |
| 16:54:58 | KeithMnemonic | nope same error with a deleted instance in cell0, | |
| 16:56:15 | mriedem | ok when i've had to debug these types of failures before i've needed to add a LOG.exception here in the API https://github.com/openstack/nova/blob/2718de6ed7c21f8ff8cf74164ae5054531fdbc30/nova/api/openstack/compute/servers.py#L331 to get a traceback | |
| 16:56:23 | KeithMnemonic | but just --deleted --limit -1 shows deleted from cell0 and cell1 so it is definitely something with the marker | |
| 16:56:25 | mriedem | KeithMnemonic: is that something you can do and then recreate to get the trace? | |
| 16:56:47 | KeithMnemonic | so this is interesting | |
| 16:57:01 | KeithMnemonic | "https://192.168.23.35:8774/v2.1/428982d4248a419a933668b6a4dd14a0/servers?name=c244e139-98fe-44eb-a971-c067954da026" will this work without deleted being passed? | |
| 16:57:18 | melwitt | I'm wondering what this get_marker_by_values is | |
| 16:57:28 | mriedem | KeithMnemonic: that's not a server name | |
| 16:57:43 | KeithMnemonic | that is what --debug shows | |
| 16:57:45 | mriedem | i'd expect you to get no results in that case | |
| 16:58:37 | mriedem | KeithMnemonic: hmm, ok so it's probably the cli trying to determine if the marker provided is a server name or id | |
| 16:58:46 | KeithMnemonic | i pass in --marker c244e139-98fe-44eb-a971-c067954da026 | |
| 16:58:57 | KeithMnemonic | no option to specify name vs ID | |
| 16:58:59 | mriedem | --marker <server>¶ The last server of the previous page. Display list of servers after marker. Display all servers if not specified. (name or ID) | |
| 16:59:56 | mriedem | that might be a red herring if osc is just trying to determine if you gave a name or id | |
| 17:00:28 | mriedem | try to recreate with the LOG.exception injection i mentioned above to see where the MarkerNotFound is coming from | |
| 17:00:43 | mriedem | you know, we might not even be getting to the api | |
| 17:00:57 | mriedem | "No server with a name or ID of 'c244e139-98fe-44eb-a971-c067954da026' exists." sounds like an osc bug | |
| 17:01:06 | mriedem | ok i bet i know, | |
| 17:01:13 | mriedem | the osc code is trying to find the marker server by name or id, | |
| 17:01:19 | mriedem | but failing to include the deleted=True filter, | |
| 17:01:32 | mriedem | so the api says "nope a server that's not deleted does not exist for that" | |
| 17:01:38 | KeithMnemonic | but should it worked without specifying deleted in the api call? | |
| 17:02:10 | mriedem | meaning can you specify a marker that is a deleted server? yes | |
| 17:02:22 | mriedem | it seems the bug is in osc though | |
| 17:03:03 | mriedem | https://github.com/openstack/python-openstackclient/blob/4.0.0/openstackclient/compute/v2/server.py#L1415 | |
| 17:03:09 | melwitt | wait, how do we know it's not including the deleted=True filter? it should show that in the --debug right | |
| 17:03:11 | mriedem | osc isn't looking for a deleted marker | |
| 17:03:44 | KeithMnemonic | it is osc, you can not do a show on a deleted sever, you get the same curl and same error | |
| 17:03:45 | mriedem | https://192.168.23.35:8774/v2.1/428982d4248a419a933668b6a4dd14a0/servers?name=c244e139-98fe-44eb-a971-c067954da026 means osc is first looking to see if the marker server is given on the command line by name | |
| 17:03:45 | melwitt | I mean, when you pass --deleted doesn't it just send deleted=True regardless? or no | |
| 17:03:50 | melwitt | oh | |
| 17:03:52 | mriedem | it's not, it gets a 404 and then tries by id, | |
| 17:04:08 | mriedem | which returns a 404 because the server is deleted and you're not including the ?deleted=true filter param | |
| 17:04:13 | KeithMnemonic | correct | |
| 17:04:23 | KeithMnemonic | but osc is called with --deleted | |
| 17:04:24 | mriedem | so it's a shell bug | |
| 17:04:30 | KeithMnemonic | it is just not passed to the api call | |
| 17:04:35 | mriedem | right, it's a bug | |
| 17:04:42 | KeithMnemonic | ok against osc | |
| 17:04:44 | mriedem | yes | |
| 17:05:17 | KeithMnemonic | i can file another today. i will reference this discussion and some debug output. | |
| 17:05:20 | KeithMnemonic | is that ok | |
| 17:05:31 | mriedem | sure | |
| 17:05:36 | mriedem | that goes against osc in storyboard though | |
| 17:05:52 | mriedem | where it will languish for 1000 years | |
| 17:06:26 | KeithMnemonic | ok let me test on something newer to see if it still exists is train for example | |
| 17:06:34 | mriedem | that code i linked above is train | |
| 17:06:38 | mriedem | so it's there | |
| 17:07:35 | mriedem | https://github.com/openstack/osc-lib/blob/1.14.1/osc_lib/utils/__init__.py#L169 | |
| 17:07:49 | mriedem | looks like the fix is relatively simple: | |
| 17:08:12 | mriedem | utils.find_resource(compute_client.servers, parsed_args.marker, deleted=parsed_args.deleted) | |
| 17:10:48 | mriedem | KeithMnemonic: feel free to have suse send me a check for the openstack support anytime :) | |
| 17:12:31 | KeithMnemonic | you know that is a sore point these days | |
| 17:13:07 | dansmith | speaking of that, | |
| 17:13:22 | dansmith | did aspiers ever show up and/or has anyone else been able to resolve the SEV issue we released with train? | |
| 17:13:45 | KeithMnemonic | AFAIK, aspiers is till with SUSE but not sure in what capacity | |
| 17:14:10 | KeithMnemonic | mriedem since that fix is simple, can i take a stab at it? | |
| 17:14:16 | mriedem | KeithMnemonic: go nuts | |
| 17:14:20 | KeithMnemonic | ok thanks | |
| 17:17:06 | dansmith | supposedly the SEV thing was resolvable by someone else, but has that happened? | |
| 17:18:24 | KeithMnemonic | someone at SUSE? dansmith | |
| 17:18:29 | mriedem | you mean boris? https://review.opendev.org/#/c/684825/ | |
| 17:18:32 | mriedem | he's also suse | |
| 17:18:46 | KeithMnemonic | he is still on openstack as well | |
| 17:29:05 | openstackgerrit | Merged openstack/nova master: Adds view builders for keypairs controller https://review.opendev.org/347289 | |
| 17:34:19 | dansmith | KeithMnemonic: no, I meant assuming nobody from SUSE is going to fix it, there was an assertion from the libvirty people that they could resolve it too, I just haven't heard that it happened | |
| 17:34:20 | dansmith | efried: ? | |
| 17:35:38 | KeithMnemonic | ok | |
| 18:00:45 | mriedem | dansmith: so i'd be +2 on https://review.opendev.org/#/c/687348/ except, | |
| 18:01:05 | mriedem | i think we should probably drop the dependency on the osc command for now because the horizon thing holding up novaclient 16.0.0 in upper-constraints is not going well | |
| 18:01:27 | mriedem | could use the nova cli in the docs for now and leave a todo in there to replace with osc later | |
| 18:01:52 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 18:01:53 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522 | |
| 18:01:53 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 18:02:08 | dansmith | mriedem: there's no real rush to merge that is there? | |
| 18:02:16 | mriedem | the docs? | |
| 18:02:28 | mriedem | i'd like to merge it sooner than later so i don't lose the context on it | |
| 18:02:35 | mriedem | especially since we have a nova cli for the thing | |
| 18:04:13 | dansmith | okay | |
| 18:10:19 | efried | dansmith: I talked to aspiers a couple weeks ago and he said he was going to try to fix the thing regardless of what SUSE does. | |