| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-22 | |||
| 20:54:20 | melwitt | mriedem: the subnode is nova-compute only, right? if so, then it would make sense to *not* run the proxy service there and it might just be that we have a wrong assumption in the conditional that the service will necessarily be running if we're doing tls between the computes and the proxy? | |
| 20:54:46 | mriedem | yeah subnode is nova-compute only | |
| 20:54:53 | mriedem | no n-novnc service running on the subnode | |
| 20:55:07 | melwitt | and maybe the intermittent is because depending on whether the instance being tested for vnc lands on the subnode or not? | |
| 20:55:13 | mriedem | yup i think so | |
| 20:56:00 | melwitt | yeah, so it sounds like we need to use some other condition for whether to do the stuff, not the presence of the tls proxy service thing | |
| 20:57:39 | mriedem | ok so maybe for now i'll rev this patch, comment out the usage of NOVA_CONSOLE_PROXY_COMPUTE_TLS since it doesn't work for multinode jobs, and see if that at least gets this patch moving forward | |
| 21:00:14 | melwitt | ok. and I can work on a change in devstack to make it work with multinode, and then stack a change on top of yours that Depends-On it, to see if it clears up the problem | |
| 21:00:47 | melwitt | or sorry, I'll just do the devstack change and then you can uncomment and Depends-On to test | |
| 21:01:21 | mriedem | sure that works, thanks | |
| 21:01:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | |
| 21:11:31 | KeithMnemonic | so what i can tell is first it calls /server/detail?deleted=True and this returns a good list, then the next call is that same but with &marker=<id of last server> and it loops here | |
| 21:12:43 | KeithMnemonic | so something about the marker. if i try and call it using curl i get a 401 but my guess is you that is not a valid test | |
| 21:12:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Log CellTimeout traceback in scatter_gather_cells" https://review.opendev.org/690417 | |
| 21:13:05 | mriedem | KeithMnemonic: you're getting a 401 with curl b/c you're not providing a token | |
| 21:13:11 | KeithMnemonic | no i am | |
| 21:13:17 | KeithMnemonic | it only 401 with &marker | |
| 21:13:28 | KeithMnemonic | with just ?delete=True it is fine | |
| 21:13:38 | mriedem | 401 is an auth issue so that doesn't make much sense | |
| 21:15:31 | KeithMnemonic | http://paste.openstack.org/show/785488/ | |
| 21:15:49 | KeithMnemonic | you can see something with adding &marker, but it could be my test is wrong | |
| 21:15:58 | KeithMnemonic | i was just trying to curl what is was looping on | |
| 21:16:24 | KeithMnemonic | the --debug show it loops once it adds on &marker=<last deleted UUID> | |
| 21:19:24 | KeithMnemonic | https://bugs.launchpad.net/nova/+bug/1773945 | |
| 21:19:24 | openstack | Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,Incomplete] | |
| 21:20:22 | mriedem | i think i see the problem | |
| 21:20:30 | KeithMnemonic | not quite my issue , but something with markers | |
| 21:20:57 | mriedem | https://review.opendev.org/#/c/527564/1/nova/compute/api.py | |
| 21:21:14 | mriedem | objects.BuildRequestList.get_by_filters returns an empty list if filtering on deleted=True, | |
| 21:21:36 | mriedem | the code assumes that if there was a marker and we got a build request list (not MarkerNotFound), we found our marker and can set marker=None since we won't find it in a cell, | |
| 21:21:41 | mriedem | but that's not the case here | |
| 21:22:00 | mriedem | that goes beyond rocky https://review.opendev.org/#/q/I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab | |
| 21:22:53 | mriedem | https://review.opendev.org/#/c/527564/1/nova/objects/build_request.py@376 | |
| 21:24:10 | mriedem | KeithMnemonic: in your reproduce do you have any deleted servers in the db? | |
| 21:24:15 | KeithMnemonic | especially since i only have 18 deleted, it seems it should not need the marker right? | |
| 21:24:19 | mriedem | you must otherwise you wouldn't get a marker | |
| 21:26:28 | melwitt | maybe the deleted instances are located across separate cells? | |
| 21:26:44 | mriedem | i meant i see the reason for the infinite loop | |
| 21:26:47 | mriedem | KeithMnemonic: did you open a bug? | |
| 21:27:18 | KeithMnemonic | not yet, was trying to get more debug to make the bug report useful | |
| 21:27:27 | KeithMnemonic | that is when i found it looped on marker | |
| 21:27:41 | melwitt | I was talking about why a marker when only 18 deleted? | |
| 21:27:54 | melwitt | I dunno | |
| 21:27:56 | mriedem | because limit is in the request i think | |
| 21:28:01 | mriedem | limit == paging | |
| 21:28:22 | KeithMnemonic | correct --limit -1 | |
| 21:28:23 | melwitt | oh ok | |
| 21:29:04 | KeithMnemonic | mriedem, ok i can open the bug, do you need anything specific in it or you have a handle on the issue? | |
| 21:29:16 | mriedem | the paste is useful and recreate steps | |
| 21:29:35 | KeithMnemonic | ok let me add all of that | |
| 21:31:08 | mriedem | nice to know that regressions i introduced in queens are coming back to find me now | |
| 21:32:13 | KeithMnemonic | lol | |
| 21:34:39 | efried | mriedem: serves you right for not giving your features enough soak time. | |
| 21:40:19 | mriedem | heh | |
| 21:40:32 | mriedem | doesn't really fit here, but i can see that's been smoldering in your pocket for awhile | |
| 21:42:14 | mriedem | i can never remember if you specify marker if the marker instance comes back as the first item in the list | |
| 21:42:19 | mriedem | or if it's everything after the marker | |
| 21:43:13 | mriedem | aha, ":param marker: the last item of the previous page; we returns the next | |
| 21:43:13 | mriedem | results after this value." | |
| 21:52:58 | mriedem | KeithMnemonic: ok i've got a recreate functional test ready to push up whenever you have this bug filed | |
| 22:00:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug xxxxxx https://review.opendev.org/690428 | |
| 22:00:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix listing deleted servers with a marker https://review.opendev.org/690429 | |
| 22:06:56 | mriedem | ok well there is the reproduce and fix ^, will update once i have a bug id | |
| #openstack-nova - 2019-10-23 | |||
| 00:39:18 | openstackgerrit | melanie witt proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | |
| 02:27:18 | KeithMnemonic | mriedem here is the bug report https://bugs.launchpad.net/nova/+bug/1849409 | |
| 02:27:18 | openstack | Launchpad bug 1849409 in OpenStack Compute (nova) "openstack server list --deleted --limit -1 hangs" [Undecided,New] | |
| 02:29:26 | KeithMnemonic | let me know if you need anything changed/added to the bug report | |
| 02:37:30 | KeithMnemonic | mriedem, i test that fix in one of my envs (pike) and it does resolve the issue | |
| 02:39:50 | KeithMnemonic | mriedem how far can we backport it, rocky, queens but not pike? | |
| 02:57:50 | openstackgerrit | melanie witt proposed openstack/nova master: Add new policy rule for viewing host status UNKNOWN https://review.opendev.org/679181 | |
| 04:09:43 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 04:09:44 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 04:09:44 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522 | |
| 07:18:36 | bauzas | good morning Nova | |
| 07:21:26 | openstackgerrit | Daniel Pawlik proposed openstack/nova master: Added openssh-client into bindep https://review.opendev.org/690482 | |
| 08:08:38 | openstackgerrit | Merged openstack/nova master: Remove compute compat checks for aborting queued live migrations https://review.opendev.org/688409 | |
| 08:08:44 | openstackgerrit | Merged openstack/nova stable/stein: rt: soften warning case in _remove_deleted_instances_allocations https://review.opendev.org/679519 | |
| 08:08:51 | openstackgerrit | Merged openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | |
| 08:27:53 | openstackgerrit | Merged openstack/nova master: Revert "Log CellTimeout traceback in scatter_gather_cells" https://review.opendev.org/690417 | |
| 10:10:43 | openstackgerrit | Merged openstack/nova stable/stein: Add functional test for resize crash compute restart revert https://review.opendev.org/687532 | |
| 10:50:54 | openstackgerrit | Merged openstack/nova stable/stein: Prevent init_host test to interfere with other tests https://review.opendev.org/687579 | |
| 10:55:42 | openstackgerrit | Merged openstack/nova stable/stein: Functional reproduce for bug 1833581 https://review.opendev.org/687534 | |
| 10:55:42 | openstack | bug 1833581 in OpenStack Compute (nova) stein "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 12:52:37 | openstackgerrit | Merged openstack/nova stable/stein: Error out interrupted builds https://review.opendev.org/687535 | |
| 13:01:55 | openstackgerrit | Merged openstack/nova stable/stein: Fix exception translation when creating volume https://review.opendev.org/688073 | |
| 13:02:02 | openstackgerrit | Merged openstack/nova stable/train: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/690099 | |
| 13:02:02 | openstack | bug 1849165 in OpenStack Compute (nova) train "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 13:02:05 | openstackgerrit | Merged openstack/nova stable/train: Don't populate resources for not-yet-migrated inst https://review.opendev.org/690100 | |
| 13:02:11 | openstackgerrit | Merged openstack/nova stable/stein: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686264 | |
| 13:02:17 | openstackgerrit | Merged openstack/nova stable/rocky: Fix unit of hw_rng:rate_period https://review.opendev.org/689154 | |
| 13:02:25 | openstackgerrit | Merged openstack/nova stable/rocky: Skip test_parallel_evacuate_with_server_group until fixed https://review.opendev.org/686402 | |
| 13:43:13 | KeithMnemonic | hi mriedem, short summary- bug logged, tested fix in my env and the hang does not occur. let me know if you need anything else | |
| 13:56:48 | mriedem | yup i saw thanks, just trying to fix the other test that is failing, which means the fix won't be so clean | |
| 13:57:37 | KeithMnemonic | thanks for the update, just ping me if you need anything else | |
| 13:58:38 | openstackgerrit | Merged openstack/nova stable/queens: Fix non-existent method of Mock https://review.opendev.org/676844 | |
| 14:33:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug 1849409 https://review.opendev.org/690428 | |
| 14:33:51 | openstack | bug 1849409 in OpenStack Compute (nova) "openstack server list --deleted --limit -1 hangs" [High,Triaged] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 14:33:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix listing deleted servers with a marker https://review.opendev.org/690429 | |
| 14:33:51 | mriedem | KeithMnemonic: ^ | |
| 14:34:10 | mriedem | melwitt: dansmith: ^ we should get that back to queens before it goes to extended maintenance b/c it was a regression introduced in queens | |