| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-22 | |||
| 21:15:58 | KeithMnemonic | i was just trying to curl what is was looping on | |
| 21:16:24 | KeithMnemonic | the --debug show it loops once it adds on &marker=<last deleted UUID> | |
| 21:19:24 | openstack | Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,Incomplete] | |
| 21:19:24 | KeithMnemonic | https://bugs.launchpad.net/nova/+bug/1773945 | |
| 21:20:22 | mriedem | i think i see the problem | |
| 21:20:30 | KeithMnemonic | not quite my issue , but something with markers | |
| 21:20:57 | mriedem | https://review.opendev.org/#/c/527564/1/nova/compute/api.py | |
| 21:21:14 | mriedem | objects.BuildRequestList.get_by_filters returns an empty list if filtering on deleted=True, | |
| 21:21:36 | mriedem | the code assumes that if there was a marker and we got a build request list (not MarkerNotFound), we found our marker and can set marker=None since we won't find it in a cell, | |
| 21:21:41 | mriedem | but that's not the case here | |
| 21:22:00 | mriedem | that goes beyond rocky https://review.opendev.org/#/q/I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab | |
| 21:22:53 | mriedem | https://review.opendev.org/#/c/527564/1/nova/objects/build_request.py@376 | |
| 21:24:10 | mriedem | KeithMnemonic: in your reproduce do you have any deleted servers in the db? | |
| 21:24:15 | KeithMnemonic | especially since i only have 18 deleted, it seems it should not need the marker right? | |
| 21:24:19 | mriedem | you must otherwise you wouldn't get a marker | |
| 21:26:28 | melwitt | maybe the deleted instances are located across separate cells? | |
| 21:26:44 | mriedem | i meant i see the reason for the infinite loop | |
| 21:26:47 | mriedem | KeithMnemonic: did you open a bug? | |
| 21:27:18 | KeithMnemonic | not yet, was trying to get more debug to make the bug report useful | |
| 21:27:27 | KeithMnemonic | that is when i found it looped on marker | |
| 21:27:41 | melwitt | I was talking about why a marker when only 18 deleted? | |
| 21:27:54 | melwitt | I dunno | |
| 21:27:56 | mriedem | because limit is in the request i think | |
| 21:28:01 | mriedem | limit == paging | |
| 21:28:22 | KeithMnemonic | correct --limit -1 | |
| 21:28:23 | melwitt | oh ok | |
| 21:29:04 | KeithMnemonic | mriedem, ok i can open the bug, do you need anything specific in it or you have a handle on the issue? | |
| 21:29:16 | mriedem | the paste is useful and recreate steps | |
| 21:29:35 | KeithMnemonic | ok let me add all of that | |
| 21:31:08 | mriedem | nice to know that regressions i introduced in queens are coming back to find me now | |
| 21:32:13 | KeithMnemonic | lol | |
| 21:34:39 | efried | mriedem: serves you right for not giving your features enough soak time. | |
| 21:40:19 | mriedem | heh | |
| 21:40:32 | mriedem | doesn't really fit here, but i can see that's been smoldering in your pocket for awhile | |
| 21:42:14 | mriedem | i can never remember if you specify marker if the marker instance comes back as the first item in the list | |
| 21:42:19 | mriedem | or if it's everything after the marker | |
| 21:43:13 | mriedem | results after this value." | |
| 21:43:13 | mriedem | aha, ":param marker: the last item of the previous page; we returns the next | |
| 21:52:58 | mriedem | KeithMnemonic: ok i've got a recreate functional test ready to push up whenever you have this bug filed | |
| 22:00:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug xxxxxx https://review.opendev.org/690428 | |
| 22:00:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix listing deleted servers with a marker https://review.opendev.org/690429 | |
| 22:06:56 | mriedem | ok well there is the reproduce and fix ^, will update once i have a bug id | |
| #openstack-nova - 2019-10-23 | |||
| 00:39:18 | openstackgerrit | melanie witt proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | |
| 02:27:18 | openstack | Launchpad bug 1849409 in OpenStack Compute (nova) "openstack server list --deleted --limit -1 hangs" [Undecided,New] | |
| 02:27:18 | KeithMnemonic | mriedem here is the bug report https://bugs.launchpad.net/nova/+bug/1849409 | |
| 02:29:26 | KeithMnemonic | let me know if you need anything changed/added to the bug report | |
| 02:37:30 | KeithMnemonic | mriedem, i test that fix in one of my envs (pike) and it does resolve the issue | |
| 02:39:50 | KeithMnemonic | mriedem how far can we backport it, rocky, queens but not pike? | |
| 02:57:50 | openstackgerrit | melanie witt proposed openstack/nova master: Add new policy rule for viewing host status UNKNOWN https://review.opendev.org/679181 | |
| 04:09:43 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 04:09:44 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522 | |
| 04:09:44 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 07:18:36 | bauzas | good morning Nova | |
| 07:21:26 | openstackgerrit | Daniel Pawlik proposed openstack/nova master: Added openssh-client into bindep https://review.opendev.org/690482 | |
| 08:08:38 | openstackgerrit | Merged openstack/nova master: Remove compute compat checks for aborting queued live migrations https://review.opendev.org/688409 | |
| 08:08:44 | openstackgerrit | Merged openstack/nova stable/stein: rt: soften warning case in _remove_deleted_instances_allocations https://review.opendev.org/679519 | |
| 08:08:51 | openstackgerrit | Merged openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | |
| 08:27:53 | openstackgerrit | Merged openstack/nova master: Revert "Log CellTimeout traceback in scatter_gather_cells" https://review.opendev.org/690417 | |
| 10:10:43 | openstackgerrit | Merged openstack/nova stable/stein: Add functional test for resize crash compute restart revert https://review.opendev.org/687532 | |
| 10:50:54 | openstackgerrit | Merged openstack/nova stable/stein: Prevent init_host test to interfere with other tests https://review.opendev.org/687579 | |
| 10:55:42 | openstack | bug 1833581 in OpenStack Compute (nova) stein "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 10:55:42 | openstackgerrit | Merged openstack/nova stable/stein: Functional reproduce for bug 1833581 https://review.opendev.org/687534 | |
| 12:52:37 | openstackgerrit | Merged openstack/nova stable/stein: Error out interrupted builds https://review.opendev.org/687535 | |
| 13:01:55 | openstackgerrit | Merged openstack/nova stable/stein: Fix exception translation when creating volume https://review.opendev.org/688073 | |
| 13:02:02 | openstack | bug 1849165 in OpenStack Compute (nova) train "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 13:02:02 | openstackgerrit | Merged openstack/nova stable/train: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/690099 | |
| 13:02:05 | openstackgerrit | Merged openstack/nova stable/train: Don't populate resources for not-yet-migrated inst https://review.opendev.org/690100 | |
| 13:02:11 | openstackgerrit | Merged openstack/nova stable/stein: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686264 | |
| 13:02:17 | openstackgerrit | Merged openstack/nova stable/rocky: Fix unit of hw_rng:rate_period https://review.opendev.org/689154 | |
| 13:02:25 | openstackgerrit | Merged openstack/nova stable/rocky: Skip test_parallel_evacuate_with_server_group until fixed https://review.opendev.org/686402 | |
| 13:43:13 | KeithMnemonic | hi mriedem, short summary- bug logged, tested fix in my env and the hang does not occur. let me know if you need anything else | |
| 13:56:48 | mriedem | yup i saw thanks, just trying to fix the other test that is failing, which means the fix won't be so clean | |
| 13:57:37 | KeithMnemonic | thanks for the update, just ping me if you need anything else | |
| 13:58:38 | openstackgerrit | Merged openstack/nova stable/queens: Fix non-existent method of Mock https://review.opendev.org/676844 | |
| 14:33:51 | mriedem | KeithMnemonic: ^ | |
| 14:33:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix listing deleted servers with a marker https://review.opendev.org/690429 | |
| 14:33:51 | openstack | bug 1849409 in OpenStack Compute (nova) "openstack server list --deleted --limit -1 hangs" [High,Triaged] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 14:33:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for bug 1849409 https://review.opendev.org/690428 | |
| 14:34:10 | mriedem | melwitt: dansmith: ^ we should get that back to queens before it goes to extended maintenance b/c it was a regression introduced in queens | |
| 14:47:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: [stable-only] Disable cinder-backup service in nova-next job https://review.opendev.org/690593 | |
| 14:50:34 | sean-k-mooney | mriedem: ill try and deploy the backport of that macvtap patch locally by the way and ill report back. the fact that we were seeing libvirt errors before the tempest jobs actully start makes me think that the melonox ci is haveing other issues. | |
| 14:50:57 | mriedem | ok that's what adrianc said about failures from that job on the change on master as well | |
| 14:57:57 | adrianc | sean-k-mooney, mriedem, there are probably other issues, we have deployed internally and macvtap is working | |
| 14:58:33 | sean-k-mooney | adrianc: ya that what i was assuming too but id does not hurt to triple check these things | |
| 14:59:49 | adrianc | also the issues observed are probably because the Baremetal server was not clean before running the CI job | |
| 14:59:58 | sean-k-mooney | i looked through the devstack logs and i could not see anything obviously out of the ordenary | |
| 15:00:28 | sean-k-mooney | ya that is possibel i suspect that libvirt needed to be restarted | |
| 15:00:36 | dansmith | mriedem: so we always return from getting BR, null the marker, | |
| 15:00:52 | dansmith | mriedem: which means we keep getting the top N results from the instance list because we call without the marker yeah? | |
| 15:00:56 | melwitt | mriedem: ack | |
| 15:01:25 | mriedem | dansmith: yeah | |
| 15:01:44 | mriedem | and novaclient doesn't check to see that the marker it used in the last pass is the same as the marker it's going to use in the next pass | |
| 15:01:47 | mriedem | so the cli just hangs | |
| 15:02:06 | dansmith | hangs but is hammering the api | |
| 15:02:11 | mriedem | yup | |
| 15:02:21 | dansmith | meh this doesn't seem worth fixing | |
| 15:02:34 | dansmith | hah | |
| 15:02:59 | dansmith | how has this not been a thing before? like, doesn't this break almost everyone's pagination? | |
| 15:03:39 | mriedem | i'm surprised it hasn't come up earlier, so idk, | |
| 15:03:54 | mriedem | it's a combination of using that --limit -1 semantic in the CLI and filtering on --deleted servers | |