| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-22 | |||
| 20:23:58 | mdbooth | And for the image fixture thing we could just use a hardcoded image | |
| 20:24:04 | mriedem | right now you have 2 refactor changes before the new regression test | |
| 20:24:05 | mdbooth | But lets do that in the backport | |
| 20:26:46 | melwitt | mriedem: without looking at the code first, I'd have thought "yes" because the instance would begin as SOFT_DELETED and then when the reclaim interval comes around and it's time to permadelete it, it would then become deleted != 0 as well. it would surprise me if the vm_state is cleared when that happens, but who knows | |
| 20:27:58 | mriedem | melwitt: the periodic queries the db for instances which are SOFT_DELETED and not undergoing some task_state transition on the given host, | |
| 20:28:10 | mriedem | then deletes them which changes the vm_state to DELETED and destroys the instance | |
| 20:28:26 | mriedem | so if we updated the vm_state to DELETED it shouldn't be in the db query results in the periodic | |
| 20:28:47 | mriedem | and if we didn't update that, like the update failed, then the instance won't be deleted either (deleted == 0) | |
| 20:29:17 | mriedem | the db api query will return deleted instances by default since the filter being used doesn't include deleted=False | |
| 20:29:31 | mriedem | but still, not sure how to get into deleted=id and vm_state=soft-delete at the same time | |
| 20:29:55 | mriedem | i was thinking maybe some local delete in the api wonkiness but didn't see anything obvoius | |
| 20:30:31 | mriedem | melwitt: also while i've got you, it looks like NOVA_CONSOLE_PROXY_COMPUTE_TLS in our nova-next job isn't getting honored anymore, i.e. we don't have the vencrypt config in that job | |
| 20:30:40 | melwitt | ok, it sounds like by "could have" you mean can the periodic return an instance like that. I answered it as though it were, "can we have any records in that state?" and I thought yes. so... nevermind me | |
| 20:30:49 | mriedem | because NOVA_CONSOLE_PROXY_COMPUTE_TLS=true in localrc but https://github.com/openstack/devstack/blob/1a46c898db9c16173013d95e2bc954992121077c/lib/nova#L175 | |
| 20:31:04 | melwitt | mriedem: argh again huh... ok, I'll look. I've had to fix that before in the past | |
| 20:31:26 | mriedem | i'm trying to figure out why https://review.opendev.org/#/c/683988/ is intermittenly failing novnc tests, | |
| 20:31:41 | melwitt | I'll look up what I did last time and compare to see what has changed | |
| 20:31:42 | mriedem | which you recently fixed for multinode jobs here https://review.opendev.org/#/c/675721/ | |
| 20:32:00 | mriedem | i think it's just the conversion of the nova-next job from d-g to zuulv3 | |
| 20:32:55 | mriedem | https://review.opendev.org/#/c/670196/ | |
| 20:33:12 | mriedem | d-g would pass the value through in the localrc but zuulv3 is yaml'ing it and converting True to true | |
| 20:33:37 | mriedem | yaml-ama-ding-dong | |
| 20:33:41 | melwitt | this is what I had to do last time to fix enablement of tls between nova-compute and the proxy https://review.opendev.org/643045 | |
| 20:34:05 | mriedem | we aren't even getting that far because https://review.opendev.org/#/c/643045/9/lib/nova@659 | |
| 20:34:15 | mriedem | [[ ${NOVA_CONSOLE_PROXY_COMPUTE_TLS} = "True" ]] | |
| 20:34:33 | mriedem | NOVA_CONSOLE_PROXY_COMPUTE_TLS: true | |
| 20:34:45 | melwitt | oh, you're saying that flag is not getting reflected as true? O.o | |
| 20:34:51 | KeithMnemonic | is this a valid command option, using --deleted and --limit -1. it seems to cause an infinite hang on Rocky. i did not check newwer releases yet | |
| 20:35:11 | mriedem | melwitt: hard to say because local.conf here looks OK https://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/local_conf.txt.gz#54 | |
| 20:35:30 | melwitt | hmm | |
| 20:36:14 | melwitt | KeithMnemonic: I didn't think that limit -1 was a thing. unlimited happens by omitting --limit, I thought. but regardless of that, there shouldn't be an indefinite hang, that I would call a bug | |
| 20:36:37 | mriedem | limit=-1 is a novaclient thing to page until there are no more results | |
| 20:37:02 | melwitt | oh :\ I thought it did that on its own in the absence of --limit | |
| 20:37:04 | KeithMnemonic | " --limit <num-servers> | |
| 20:37:04 | KeithMnemonic | Maximum number of servers to display. If limit equals | |
| 20:37:04 | KeithMnemonic | -1, all servers will be displayed. If limit is greater | |
| 20:37:04 | KeithMnemonic | than 'osapi_max_limit' option of Nova API, | |
| 20:37:04 | KeithMnemonic | 'osapi_max_limit' will be used instead. | |
| 20:37:05 | KeithMnemonic | " | |
| 20:37:11 | melwitt | nevermind, sorry | |
| 20:37:13 | KeithMnemonic | from the cli help | |
| 20:37:27 | KeithMnemonic | ok i can log a bug. i will see if this happens on something newer | |
| 20:37:27 | mriedem | melwitt: nope, by default limit is 1000 results in the api | |
| 20:37:36 | KeithMnemonic | i did not see anything open around this currently | |
| 20:37:52 | melwitt | mriedem: that I know, but I thought the client would auto-page by itself without need to tell it | |
| 20:37:56 | melwitt | and I was wrong | |
| 20:38:12 | KeithMnemonic | in the env i tested, i only have maybe 20 deleted at most | |
| 20:38:34 | melwitt | yeah, something is whack there, I think | |
| 20:39:35 | KeithMnemonic | let me see what debug i can get to add to the bug to help isolate it | |
| 20:39:49 | mdbooth | mriedem: Responded. I still think that's good to go. I'd prefer to make the requested changes in the backports. They're trivial enough. | |
| 20:40:47 | melwitt | KeithMnemonic: yeah run with --debug to see where it gets stuck | |
| 20:50:16 | mriedem | melwitt: oh here is the vencrypt stuff, i was looking in the wrong config file (there are only 20 nova configs in a devstack job) | |
| 20:50:17 | mriedem | https://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/etc/nova/nova_cell1_conf.txt.gz#32 | |
| 20:51:46 | mriedem | so i wonder if NOVA_CONSOLE_PROXY_COMPUTE_TLS=True just doesn't work for the subnode since we've never had a multinode job use that yet | |
| 20:52:33 | melwitt | mriedem: ah yeah. awhile back I split things at stephenfin request (and I agree was a good idea) to only config which settings were required on the proxy and compute host, respectively. so they don't just all appear in all configs like they used to back in the day | |
| 20:54:20 | melwitt | mriedem: the subnode is nova-compute only, right? if so, then it would make sense to *not* run the proxy service there and it might just be that we have a wrong assumption in the conditional that the service will necessarily be running if we're doing tls between the computes and the proxy? | |
| 20:54:46 | mriedem | yeah subnode is nova-compute only | |
| 20:54:53 | mriedem | no n-novnc service running on the subnode | |
| 20:55:07 | melwitt | and maybe the intermittent is because depending on whether the instance being tested for vnc lands on the subnode or not? | |
| 20:55:13 | mriedem | yup i think so | |
| 20:56:00 | melwitt | yeah, so it sounds like we need to use some other condition for whether to do the stuff, not the presence of the tls proxy service thing | |
| 20:57:39 | mriedem | ok so maybe for now i'll rev this patch, comment out the usage of NOVA_CONSOLE_PROXY_COMPUTE_TLS since it doesn't work for multinode jobs, and see if that at least gets this patch moving forward | |
| 21:00:14 | melwitt | ok. and I can work on a change in devstack to make it work with multinode, and then stack a change on top of yours that Depends-On it, to see if it clears up the problem | |
| 21:00:47 | melwitt | or sorry, I'll just do the devstack change and then you can uncomment and Depends-On to test | |
| 21:01:21 | mriedem | sure that works, thanks | |
| 21:01:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | |
| 21:11:31 | KeithMnemonic | so what i can tell is first it calls /server/detail?deleted=True and this returns a good list, then the next call is that same but with &marker=<id of last server> and it loops here | |
| 21:12:43 | KeithMnemonic | so something about the marker. if i try and call it using curl i get a 401 but my guess is you that is not a valid test | |
| 21:12:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Log CellTimeout traceback in scatter_gather_cells" https://review.opendev.org/690417 | |
| 21:13:05 | mriedem | KeithMnemonic: you're getting a 401 with curl b/c you're not providing a token | |
| 21:13:11 | KeithMnemonic | no i am | |
| 21:13:17 | KeithMnemonic | it only 401 with &marker | |
| 21:13:28 | KeithMnemonic | with just ?delete=True it is fine | |
| 21:13:38 | mriedem | 401 is an auth issue so that doesn't make much sense | |
| 21:15:31 | KeithMnemonic | http://paste.openstack.org/show/785488/ | |
| 21:15:49 | KeithMnemonic | you can see something with adding &marker, but it could be my test is wrong | |
| 21:15:58 | KeithMnemonic | i was just trying to curl what is was looping on | |
| 21:16:24 | KeithMnemonic | the --debug show it loops once it adds on &marker=<last deleted UUID> | |
| 21:19:24 | KeithMnemonic | https://bugs.launchpad.net/nova/+bug/1773945 | |
| 21:19:24 | openstack | Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,Incomplete] | |
| 21:20:22 | mriedem | i think i see the problem | |
| 21:20:30 | KeithMnemonic | not quite my issue , but something with markers | |
| 21:20:57 | mriedem | https://review.opendev.org/#/c/527564/1/nova/compute/api.py | |
| 21:21:14 | mriedem | objects.BuildRequestList.get_by_filters returns an empty list if filtering on deleted=True, | |
| 21:21:36 | mriedem | the code assumes that if there was a marker and we got a build request list (not MarkerNotFound), we found our marker and can set marker=None since we won't find it in a cell, | |
| 21:21:41 | mriedem | but that's not the case here | |
| 21:22:00 | mriedem | that goes beyond rocky https://review.opendev.org/#/q/I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab | |
| 21:22:53 | mriedem | https://review.opendev.org/#/c/527564/1/nova/objects/build_request.py@376 | |
| 21:24:10 | mriedem | KeithMnemonic: in your reproduce do you have any deleted servers in the db? | |
| 21:24:15 | KeithMnemonic | especially since i only have 18 deleted, it seems it should not need the marker right? | |
| 21:24:19 | mriedem | you must otherwise you wouldn't get a marker | |
| 21:26:28 | melwitt | maybe the deleted instances are located across separate cells? | |
| 21:26:44 | mriedem | i meant i see the reason for the infinite loop | |
| 21:26:47 | mriedem | KeithMnemonic: did you open a bug? | |
| 21:27:18 | KeithMnemonic | not yet, was trying to get more debug to make the bug report useful | |
| 21:27:27 | KeithMnemonic | that is when i found it looped on marker | |
| 21:27:41 | melwitt | I was talking about why a marker when only 18 deleted? | |
| 21:27:54 | melwitt | I dunno | |
| 21:27:56 | mriedem | because limit is in the request i think | |
| 21:28:01 | mriedem | limit == paging | |