Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-18
18:40:30 sean-k-mooney the only one im aware of is that you will nolonger be able to retivie the migration info since you deleted it.
18:41:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453
18:41:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963
18:41:43 mloza understand. I don't need it for auditing
18:41:45 mloza Thanks
18:41:58 mriedem don't take that as a recommendation that it's ok
18:42:08 mriedem messing with the db directly is not really a supported thing, so beware
18:42:39 mriedem especially with migration-based allocation consumers since queens i could see us (nova) relying on migration records existing to determine a type of consumer of resource allocatoins in placement
18:42:47 mriedem like in the audit command that bauzas is working on
18:42:53 mriedem mloza: ^
18:43:09 sean-k-mooney right from a downstream perspecit i think we would normally ask you to file a support exception first and assess if its safe
18:43:52 mriedem for example, let's say a migration failed and we failed to cleanup allocatoins in placement properly, then placement might be saying there are more consumers of resources than there actually are, and without some kind of audit tooling to determine that - which would use migration records - you might have a hard time sorting that all out
18:44:56 mriedem that kind of orphan issue can lead to a situation where you expect to be able to land new VMs on a host but placement filters them out saying there is no room
18:45:02 sean-k-mooney yes. our customer issue was related to FFU and failed evauctations leave evaucate migration recored in pre-migrating
18:45:31 sean-k-mooney in that case in queens we deleted there vms when the comptue agent start up
18:46:48 sean-k-mooney that said they didnt porperly check the evaucations before unfenceign the node
18:47:33 mriedem that said you didn't provide them a tool to check when it was safe to unfence the ndoe
18:47:34 mriedem *node
18:47:41 mriedem especially for the $$$ they are giving you
18:48:16 sean-k-mooney true but the db recoreds showed they used --force --host to do a nova evac to the same host
18:48:24 sean-k-mooney which is why the evacuation failed
18:49:16 sean-k-mooney you cant evacuate to the same host that you are evacuating form. we should however have an api check for that
18:49:42 sean-k-mooney this came up last week so we havent really gotten around to figuring out what needs to be done upstream yet
18:50:00 mriedem well, w/o --force the scheduler would have kicked it out
18:50:06 sean-k-mooney yep
18:50:07 mriedem as long as the compute was still disabled
18:50:19 mriedem we should probably deprecate the --force option on the evacuate cli https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-evacuate
18:50:21 sean-k-mooney if the compute was active it would have kicked it out too
18:50:39 sean-k-mooney mriedem: didnt you already do that?
18:50:53 mriedem the force parameter was deprecated in the api
18:50:58 mriedem but it's still there on earlier microversions
18:51:28 mriedem added in 2.29 and removed in 267
18:51:36 sean-k-mooney ah ok i remember talking about not supporting it in the osc version
18:51:49 mloza mriedem: I guess I hitting that orphan issue. I possibly cleared the migration records before. Now wonder I everytime I launch a new VM it goes always goes to a specific host.
18:51:49 sean-k-mooney speaking of which i should update that patch.
18:52:15 mloza What would be a remedy to this issue?
18:52:35 sean-k-mooney mloza: what release are you running
18:52:41 mloza stable/stein
18:52:57 mriedem sean-k-mooney: please tell me you ran pep8 on this https://review.opendev.org/#/c/669738/8..
18:52:58 sean-k-mooney ok i was wondering if you had the old failed builds behavior.
18:52:59 mriedem https://review.opendev.org/#/c/669738/
18:54:11 sean-k-mooney i think i did
18:54:18 sean-k-mooney i know i had to fix the odering of the imports
18:54:34 mriedem mloza: this is a work in progress but you might be able to use this https://review.opendev.org/#/c/670112/ - that would also provide valuable feedback on what it reports
18:55:38 mriedem mloza: you can use https://docs.openstack.org/osc-placement/latest/ to see what is consuming resource allocations on your nodes (resource providers)
18:55:55 mriedem i would start by investigating a particular host that you think should be available but placement is saying it's not
18:56:37 mloza will take a look
18:56:40 mriedem so using this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show with the --allocations option
18:57:02 mriedem the allocations are keyed by consumer uuid, which in the case of nova since queens can be migrations or instances
18:57:23 mriedem migrations for the source host during a migration, instances for the dest host
18:57:57 mriedem so let's say you evacuated all instances from a down host but placement is still reporting allocations against that host - those allocation consumers are likely migration records
18:57:59 mriedem and were orphaned
18:58:33 mriedem there are a couple of related bugs in this patch https://review.opendev.org/#/c/678100/
19:18:35 mriedem melwitt: how are you feeling about this? https://review.opendev.org/#/c/541420/ i split the refactor out and without the compat code for old cinder it's quite a bit simpler than the last time you looked.
20:03:55 KeithMnemonic1 I have a customer running into an odd issue with multipath during migration (Pike). To make a long story short the LUNS assignments are getting mixed up somewhere. geguileo mentioned it possibly could be this https://review.opendev.org/#/c/551302/ I did a quick look at cherry-picking and it seems there are a ton of dependencies. Does that seems accurate or does someone have some magic to cherry-pick it to
20:03:55 KeithMnemonic1 pike?
20:15:29 KeithMnemonic1 mriedem, melwitt any thoughts on my question above
20:16:27 mriedem coincidentally that came up yesterday on pike https://review.opendev.org/#/c/670016/
20:16:44 mriedem ^ will not fix your issue though
20:17:33 mriedem looking at the conflicts on the queens backport i'm not surprised that there would be conflicts going to pike, but w/o doing it myself i do'nt know how bad it is,
20:17:45 mriedem red hat probably doesn't care to backport that since i don't think they are supporting pike
20:23:47 mriedem efried_pto: are you going to be running the meeting tomorrow morning or should someone else?
20:39:26 KeithMnemonic1 mriedem i was using aspiers git-deps and it was a long list. I am not experienced enough to know if there are any shortcuts
20:48:30 KeithMnemonic1 I was looking to confirm with someone if my inital look was correct (that it is too cumbersome) or if it is not that bad
20:50:10 mriedem i can peek in a bit
20:51:07 melwitt mriedem: I dunno, haven't looked at it in a long time. will go through it today
20:56:37 melwitt KeithMnemonic1, mriedem: I can confirm that we haven't backported https://review.opendev.org/#/c/551302/ beyond queens downstream either, so I also don't know what to expect as far as how gnarly the backport will be without actually trying to do it
20:58:12 KeithMnemonic1 I took a look and it seemed gnarly but was hoping for a second opinion
21:00:13 mriedem is this a new failure in building docs locally?
21:00:13 mriedem WARNING: RSVG converter command 'rsvg-convert' cannot be run. Check the rsvg_converter_bin setting
21:00:59 KeithMnemonic1 melwitt a plug for aspiers this is what I used to gauge the effort https://github.com/aspiers/git-deps
21:04:31 KeithMnemonic1 but that pulls in every dependency and i am not experienced enough to know if any could be skipped
21:05:43 mriedem ah i see https://github.com/openstack/nova/commit/16b9486bf7e91bfd5dc48297cee9f54b49156c93
21:15:19 mriedem we need librsvg2-bin in bindep.txt
21:19:33 mriedem KeithMnemonic1: do you have a paste of the dependencies? i wouldn't be surprised if a bunch of the conflicts are due to like mox removal patches
21:28:11 mriedem melwitt: if you have a fedora or centos system available can you confirm that librsvg2-tools provides the rsvg-convert command?
21:28:53 mriedem which is i think just: yum provides rsvg-convert
21:28:53 mriedem right?
21:29:13 melwitt let me check
21:31:23 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Re-propose cross-cell-resize spec for Ussuri https://review.opendev.org/683002
21:33:15 melwitt mriedem: you are correct, comes from librsvg2-tools
21:33:39 mriedem cool, thanks. that's what cinder had in their bindep but i've only got ubuntu and that's all that zuul uses for the pdf jobs
21:33:43 melwitt looked on centos7.3
21:34:05 openstackgerrit Matt Riedemann proposed openstack/nova master: Add librsvg2* to bindep https://review.opendev.org/683003
21:34:55 mriedem looks like we just had a zuul restart
21:40:18 mriedem and it's busted, "Unable to freeze job graph: 'dict_keys' object does not support indexing"
21:40:28 mriedem on that note, looks like i'm out of there for the day
21:49:00 mriedem_afk KeithMnemonic1: quickly glancing at that pike backport from queens, there are a bunch of conflicts but the big ones look like they are due to the multiattach volume support added in queens but those aren't functional dependencies, so i don't think those would actually impact backporting the fix to pike except it makes the backport harder
22:34:46 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020
22:34:54 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020
22:36:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027
22:36:30 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027
22:39:45 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031
22:39:58 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031
22:41:13 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299
22:41:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299
22:42:06 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344
22:42:18 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344
22:43:30 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673
22:43:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673

Earlier   Later