Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-23
13:43:13 KeithMnemonic hi mriedem, short summary- bug logged, tested fix in my env and the hang does not occur. let me know if you need anything else
13:56:48 mriedem yup i saw thanks, just trying to fix the other test that is failing, which means the fix won't be so clean
13:57:37 KeithMnemonic thanks for the update, just ping me if you need anything else
13:58:38 openstackgerrit Merged openstack/nova stable/queens: Fix non-existent method of Mock https://review.opendev.org/676844
14:33:51 mriedem KeithMnemonic: ^
14:33:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix listing deleted servers with a marker https://review.opendev.org/690429
14:33:51 openstack bug 1849409 in OpenStack Compute (nova) "openstack server list --deleted --limit -1 hangs" [High,Triaged] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem)
14:33:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional regression test for bug 1849409 https://review.opendev.org/690428
14:34:10 mriedem melwitt: dansmith: ^ we should get that back to queens before it goes to extended maintenance b/c it was a regression introduced in queens
14:47:20 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: [stable-only] Disable cinder-backup service in nova-next job https://review.opendev.org/690593
14:50:34 sean-k-mooney mriedem: ill try and deploy the backport of that macvtap patch locally by the way and ill report back. the fact that we were seeing libvirt errors before the tempest jobs actully start makes me think that the melonox ci is haveing other issues.
14:50:57 mriedem ok that's what adrianc said about failures from that job on the change on master as well
14:57:57 adrianc sean-k-mooney, mriedem, there are probably other issues, we have deployed internally and macvtap is working
14:58:33 sean-k-mooney adrianc: ya that what i was assuming too but id does not hurt to triple check these things
14:59:49 adrianc also the issues observed are probably because the Baremetal server was not clean before running the CI job
14:59:58 sean-k-mooney i looked through the devstack logs and i could not see anything obviously out of the ordenary
15:00:28 sean-k-mooney ya that is possibel i suspect that libvirt needed to be restarted
15:00:36 dansmith mriedem: so we always return from getting BR, null the marker,
15:00:52 dansmith mriedem: which means we keep getting the top N results from the instance list because we call without the marker yeah?
15:00:56 melwitt mriedem: ack
15:01:25 mriedem dansmith: yeah
15:01:44 mriedem and novaclient doesn't check to see that the marker it used in the last pass is the same as the marker it's going to use in the next pass
15:01:47 mriedem so the cli just hangs
15:02:06 dansmith hangs but is hammering the api
15:02:11 mriedem yup
15:02:21 dansmith meh this doesn't seem worth fixing
15:02:34 dansmith hah
15:02:59 dansmith how has this not been a thing before? like, doesn't this break almost everyone's pagination?
15:03:39 mriedem i'm surprised it hasn't come up earlier, so idk,
15:03:54 mriedem it's a combination of using that --limit -1 semantic in the CLI and filtering on --deleted servers
15:04:27 mriedem maybe admins don't use --limit -1 often, idk
15:04:51 dansmith oh right right, forgot about the deleted thing
15:06:43 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: note the need to configure cinder auth in reclaim_instance_interval https://review.opendev.org/690602
15:17:04 openstackgerrit Adrian Chiris proposed openstack/nova stable/train: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/690126
15:25:47 mriedem gibi_off: looks like we need https://review.opendev.org/#/c/687550/ updated to keep that series moving,
15:25:54 mriedem elod: ^ if gibi is out maybe you can do that?
15:26:31 mriedem doesn't have to be today since i know it's late in the day there, but would be good to wrap that up tomorrow
15:28:15 openstackgerrit Matt Riedemann proposed openstack/nova stable/train: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/690126
15:30:11 openstackgerrit Merged openstack/python-novaclient master: Switch to Ussuri jobs https://review.opendev.org/690013
15:30:11 mriedem need another stable core on ^, that's a fix for a regression in train
15:30:43 mriedem efried: you'll like this https://review.opendev.org/#/q/owner:kangyufei+status:open
15:34:32 mriedem lyarwood: the stein version of this is merged now https://review.opendev.org/#/c/686292/
15:35:11 lyarwood mriedem: ack thanks,
15:35:24 mriedem dansmith: mind hitting this in stable/rocky? https://review.opendev.org/#/c/686276/
15:36:14 dansmith yar
15:57:59 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: rt: only map compute node if we created it https://review.opendev.org/676285
16:02:44 lyarwood mriedem: https://review.opendev.org/#/q/topic:bug/1843639 - would you mind taking a swing at this if you have time this week btw? The functional test might be a little over the top for a chance like this but it felt like the best way of verifying this at the time.
16:02:55 lyarwood change*
16:05:26 mriedem i see that depends on https://review.opendev.org/#/c/686832/ which has had a -1 sitting for awhile
16:05:30 mriedem is that going to be addressed?
16:06:44 lyarwood mriedem: I was going to talk to artom about that today once he's around
16:06:56 lyarwood mriedem: I'm happy to address it if he doesn't have time
16:11:55 mriedem lyarwood: ok i'll get out the red pen for your functional test
16:12:53 lyarwood thanks
16:13:31 mriedem man there is a lot of mocking in this func test
16:13:34 mriedem usually a red flag
16:16:20 lyarwood yeah given I'm trying to get to post_live_migration within the Libvirt driver itself I had to mock out the initial checks and migration itself
16:17:06 mriedem i'm not sure the functional test is worth it honestly,
16:17:14 mriedem especially given the simple nature of the fix in the driver
16:18:12 mriedem trying to backport that functional test is also going to be a nightmare
16:18:17 lyarwood yeah true, I started writing this while triaging the original downstream bug as a way of proving an exception from post_live_migration would leave the instance in an ERROR state pointing at the wrong host
16:18:18 mriedem well, at least a bad dream
16:18:59 mriedem that's possible without using and mocking the shit out of the libvirt driver here,
16:19:21 mriedem i.e. you could have a much much simpler functional test that just stubs the driver.post_live_migration method to raise some exception
16:19:30 mriedem bingo bango you cut out 80% of the gross of that test
16:20:03 lyarwood well that's one mock gone
16:20:08 efried mriedem: somebody had to do it :P
16:20:30 lyarwood okay maybe two
16:21:31 lyarwood that wouldn't even work anyway
16:21:42 mriedem lyarwood: i left a comment
16:21:45 lyarwood the pre migration checks are a PITA to workaround in a functional test sense
16:21:57 mriedem we have functional tests with the fake driver that all handle live migratoin
16:22:07 mriedem the only thing you have to do is stub out driver.post_live_migration to raise processutils.ProcessExecutionError
16:22:36 mriedem so if you made this much simpler and more generic i'd say cool with the functional test - and it removes your dep on artom's change - otherwise i'd say drop it
16:22:49 lyarwood that's only useful if I moved the fix up into the compute layer and out of the Libvirt driver
16:23:21 mriedem true, in the libvirt driver fix you'd have to just remove the stub which would be dumb
16:23:35 lyarwood I don't mind going that way
16:24:03 lyarwood but I was worried that other virt drivers might not want that behaviour tbh
16:26:20 mriedem i'd just drop the functional test - you've at least proven the issue with it so we can be comfortable with the driver fix on top
16:26:49 lyarwood yup that's fine, thanks for the review!
16:27:08 mriedem and now artom can go back to watching antiques roadshow
16:39:00 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/682622
16:40:51 melwitt mriedem: I dunno if you saw, but I proposed a devstack change to fix the vnc + tls thing https://review.opendev.org/690469
16:42:38 KeithMnemonic mriedem, would you expect --deleted --marker to work or is --marker only for active instances
16:43:03 mriedem melwitt: ooo cool
16:43:17 mriedem KeithMnemonic: i would expect it to work
16:43:36 mriedem KeithMnemonic: if you have 20 deleted instances and limit to 10 per page, then you'd get a marker
16:44:35 KeithMnemonic can you look at this., this is with your latest patch http://paste.openstack.org/show/785528/
16:46:06 KeithMnemonic even with --limit -1 you get the same message
16:46:30 mriedem hmm, we do look up deleted instances for markers so it's not that https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L2283
16:46:59 mriedem which cell is c244e139-98fe-44eb-a971-c067954da026 in ?
16:47:03 mriedem cell1 or cell0?
16:47:16 KeithMnemonic for not deleted it works as expected. let me check one sec
16:49:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988
16:51:35 KeithMnemonic cell1 it seems
16:52:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988
16:52:52 mriedem hmm, i wonder if there is something going on with the multi-cell instance listing stuff that has some marker logic in it
16:53:08 KeithMnemonic let me try it with a market in cell0
16:53:13 KeithMnemonic i have one instance there
16:53:16 melwitt mriedem: uh oh https://github.com/openstack/nova/blob/master/nova/compute/instance_list.py#L59

Earlier   Later