Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-22
19:37:57 dansmith lol
19:50:31 mriedem unfortunately i still have to -1 them
19:54:07 mriedem is there some way we could have an instance that is both deleted (instance.deleted != 0) and SOFT_DELETED? (instance.vm_state == 'soft-delete')?
20:18:17 openstackgerrit Matthew Booth proposed openstack/nova master: Add new simple test base for functional tests https://review.opendev.org/689186
20:18:17 openstackgerrit Matthew Booth proposed openstack/nova master: Functional test for UnexpectedDeletingTaskStateError https://review.opendev.org/689278
20:18:18 openstackgerrit Matthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks https://review.opendev.org/663382
20:18:18 openstackgerrit Matthew Booth proposed openstack/nova master: Fix incorrect vm and task state after build failure race https://review.opendev.org/689388
20:18:19 openstackgerrit Matthew Booth proposed openstack/nova master: Make image fixtures simpler to reference https://review.opendev.org/690403
20:19:55 mdbooth mriedem: https://review.opendev.org/#/c/689186/3/nova/tests/functional/integrated_helpers.py is what I did to ProviderUsageBaseTestCase
20:20:42 mdbooth tl;dr renamed it to SimpleTestBase and put all the provider test specific support code in a ProviderUsageBaseTestCase subclass
20:21:14 mdbooth Then I subclassed SimpleTestBase, which is clean without the provider stuff
20:22:58 mriedem i don't want to backport any of those refactors
20:23:03 mriedem i thought i was pretty clear about that
20:23:13 mdbooth I can't explain the locking thing I hit, btw, but it made no sense, went away when I looked harder, and never came back. I assume it was PEBKAC and I probably had an unsaved editor window open or something.
20:23:19 mdbooth mriedem: Sure, if you like.
20:23:48 mdbooth mriedem: For the backport we could just use ProviderUsageBaseTestCase
20:23:54 mriedem yes, please
20:23:58 mdbooth And for the image fixture thing we could just use a hardcoded image
20:24:04 mriedem right now you have 2 refactor changes before the new regression test
20:24:05 mdbooth But lets do that in the backport
20:26:46 melwitt mriedem: without looking at the code first, I'd have thought "yes" because the instance would begin as SOFT_DELETED and then when the reclaim interval comes around and it's time to permadelete it, it would then become deleted != 0 as well. it would surprise me if the vm_state is cleared when that happens, but who knows
20:27:58 mriedem melwitt: the periodic queries the db for instances which are SOFT_DELETED and not undergoing some task_state transition on the given host,
20:28:10 mriedem then deletes them which changes the vm_state to DELETED and destroys the instance
20:28:26 mriedem so if we updated the vm_state to DELETED it shouldn't be in the db query results in the periodic
20:28:47 mriedem and if we didn't update that, like the update failed, then the instance won't be deleted either (deleted == 0)
20:29:17 mriedem the db api query will return deleted instances by default since the filter being used doesn't include deleted=False
20:29:31 mriedem but still, not sure how to get into deleted=id and vm_state=soft-delete at the same time
20:29:55 mriedem i was thinking maybe some local delete in the api wonkiness but didn't see anything obvoius
20:30:31 mriedem melwitt: also while i've got you, it looks like NOVA_CONSOLE_PROXY_COMPUTE_TLS in our nova-next job isn't getting honored anymore, i.e. we don't have the vencrypt config in that job
20:30:40 melwitt ok, it sounds like by "could have" you mean can the periodic return an instance like that. I answered it as though it were, "can we have any records in that state?" and I thought yes. so... nevermind me
20:30:49 mriedem because NOVA_CONSOLE_PROXY_COMPUTE_TLS=true in localrc but https://github.com/openstack/devstack/blob/1a46c898db9c16173013d95e2bc954992121077c/lib/nova#L175
20:31:04 melwitt mriedem: argh again huh... ok, I'll look. I've had to fix that before in the past
20:31:26 mriedem i'm trying to figure out why https://review.opendev.org/#/c/683988/ is intermittenly failing novnc tests,
20:31:41 melwitt I'll look up what I did last time and compare to see what has changed
20:31:42 mriedem which you recently fixed for multinode jobs here https://review.opendev.org/#/c/675721/
20:32:00 mriedem i think it's just the conversion of the nova-next job from d-g to zuulv3
20:32:55 mriedem https://review.opendev.org/#/c/670196/
20:33:12 mriedem d-g would pass the value through in the localrc but zuulv3 is yaml'ing it and converting True to true
20:33:37 mriedem yaml-ama-ding-dong
20:33:41 melwitt this is what I had to do last time to fix enablement of tls between nova-compute and the proxy https://review.opendev.org/643045
20:34:05 mriedem we aren't even getting that far because https://review.opendev.org/#/c/643045/9/lib/nova@659
20:34:15 mriedem [[ ${NOVA_CONSOLE_PROXY_COMPUTE_TLS} = "True" ]]
20:34:33 mriedem NOVA_CONSOLE_PROXY_COMPUTE_TLS: true
20:34:45 melwitt oh, you're saying that flag is not getting reflected as true? O.o
20:34:51 KeithMnemonic is this a valid command option, using --deleted and --limit -1. it seems to cause an infinite hang on Rocky. i did not check newwer releases yet
20:35:11 mriedem melwitt: hard to say because local.conf here looks OK https://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/local_conf.txt.gz#54
20:35:30 melwitt hmm
20:36:14 melwitt KeithMnemonic: I didn't think that limit -1 was a thing. unlimited happens by omitting --limit, I thought. but regardless of that, there shouldn't be an indefinite hang, that I would call a bug
20:36:37 mriedem limit=-1 is a novaclient thing to page until there are no more results
20:37:02 melwitt oh :\ I thought it did that on its own in the absence of --limit
20:37:04 KeithMnemonic " --limit <num-servers>
20:37:04 KeithMnemonic Maximum number of servers to display. If limit equals
20:37:04 KeithMnemonic -1, all servers will be displayed. If limit is greater
20:37:04 KeithMnemonic than 'osapi_max_limit' option of Nova API,
20:37:04 KeithMnemonic 'osapi_max_limit' will be used instead.
20:37:05 KeithMnemonic "
20:37:11 melwitt nevermind, sorry
20:37:13 KeithMnemonic from the cli help
20:37:27 KeithMnemonic ok i can log a bug. i will see if this happens on something newer
20:37:27 mriedem melwitt: nope, by default limit is 1000 results in the api
20:37:36 KeithMnemonic i did not see anything open around this currently
20:37:52 melwitt mriedem: that I know, but I thought the client would auto-page by itself without need to tell it
20:37:56 melwitt and I was wrong
20:38:12 KeithMnemonic in the env i tested, i only have maybe 20 deleted at most
20:38:34 melwitt yeah, something is whack there, I think
20:39:35 KeithMnemonic let me see what debug i can get to add to the bug to help isolate it
20:39:49 mdbooth mriedem: Responded. I still think that's good to go. I'd prefer to make the requested changes in the backports. They're trivial enough.
20:40:47 melwitt KeithMnemonic: yeah run with --debug to see where it gets stuck
20:50:16 mriedem melwitt: oh here is the vencrypt stuff, i was looking in the wrong config file (there are only 20 nova configs in a devstack job)
20:50:17 mriedem https://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/etc/nova/nova_cell1_conf.txt.gz#32
20:51:46 mriedem so i wonder if NOVA_CONSOLE_PROXY_COMPUTE_TLS=True just doesn't work for the subnode since we've never had a multinode job use that yet
20:52:33 melwitt mriedem: ah yeah. awhile back I split things at stephenfin request (and I agree was a good idea) to only config which settings were required on the proxy and compute host, respectively. so they don't just all appear in all configs like they used to back in the day
20:54:20 melwitt mriedem: the subnode is nova-compute only, right? if so, then it would make sense to *not* run the proxy service there and it might just be that we have a wrong assumption in the conditional that the service will necessarily be running if we're doing tls between the computes and the proxy?
20:54:46 mriedem yeah subnode is nova-compute only
20:54:53 mriedem no n-novnc service running on the subnode
20:55:07 melwitt and maybe the intermittent is because depending on whether the instance being tested for vnc lands on the subnode or not?
20:55:13 mriedem yup i think so
20:56:00 melwitt yeah, so it sounds like we need to use some other condition for whether to do the stuff, not the presence of the tls proxy service thing
20:57:39 mriedem ok so maybe for now i'll rev this patch, comment out the usage of NOVA_CONSOLE_PROXY_COMPUTE_TLS since it doesn't work for multinode jobs, and see if that at least gets this patch moving forward
21:00:14 melwitt ok. and I can work on a change in devstack to make it work with multinode, and then stack a change on top of yours that Depends-On it, to see if it clears up the problem
21:00:47 melwitt or sorry, I'll just do the devstack change and then you can uncomment and Depends-On to test
21:01:21 mriedem sure that works, thanks
21:01:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988
21:11:31 KeithMnemonic so what i can tell is first it calls /server/detail?deleted=True and this returns a good list, then the next call is that same but with &marker=<id of last server> and it loops here
21:12:43 KeithMnemonic so something about the marker. if i try and call it using curl i get a 401 but my guess is you that is not a valid test
21:12:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert "Log CellTimeout traceback in scatter_gather_cells" https://review.opendev.org/690417
21:13:05 mriedem KeithMnemonic: you're getting a 401 with curl b/c you're not providing a token
21:13:11 KeithMnemonic no i am
21:13:17 KeithMnemonic it only 401 with &marker
21:13:28 KeithMnemonic with just ?delete=True it is fine
21:13:38 mriedem 401 is an auth issue so that doesn't make much sense
21:15:31 KeithMnemonic http://paste.openstack.org/show/785488/
21:15:49 KeithMnemonic you can see something with adding &marker, but it could be my test is wrong
21:15:58 KeithMnemonic i was just trying to curl what is was looping on
21:16:24 KeithMnemonic the --debug show it loops once it adds on &marker=<last deleted UUID>
21:19:24 KeithMnemonic https://bugs.launchpad.net/nova/+bug/1773945
21:19:24 openstack Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,Incomplete]
21:20:22 mriedem i think i see the problem
21:20:30 KeithMnemonic not quite my issue , but something with markers
21:20:57 mriedem https://review.opendev.org/#/c/527564/1/nova/compute/api.py

Earlier   Later