| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-18 | |||
| 18:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453 | |
| 18:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963 | |
| 18:41:43 | mloza | understand. I don't need it for auditing | |
| 18:41:45 | mloza | Thanks | |
| 18:41:58 | mriedem | don't take that as a recommendation that it's ok | |
| 18:42:08 | mriedem | messing with the db directly is not really a supported thing, so beware | |
| 18:42:39 | mriedem | especially with migration-based allocation consumers since queens i could see us (nova) relying on migration records existing to determine a type of consumer of resource allocatoins in placement | |
| 18:42:47 | mriedem | like in the audit command that bauzas is working on | |
| 18:42:53 | mriedem | mloza: ^ | |
| 18:43:09 | sean-k-mooney | right from a downstream perspecit i think we would normally ask you to file a support exception first and assess if its safe | |
| 18:43:52 | mriedem | for example, let's say a migration failed and we failed to cleanup allocatoins in placement properly, then placement might be saying there are more consumers of resources than there actually are, and without some kind of audit tooling to determine that - which would use migration records - you might have a hard time sorting that all out | |
| 18:44:56 | mriedem | that kind of orphan issue can lead to a situation where you expect to be able to land new VMs on a host but placement filters them out saying there is no room | |
| 18:45:02 | sean-k-mooney | yes. our customer issue was related to FFU and failed evauctations leave evaucate migration recored in pre-migrating | |
| 18:45:31 | sean-k-mooney | in that case in queens we deleted there vms when the comptue agent start up | |
| 18:46:48 | sean-k-mooney | that said they didnt porperly check the evaucations before unfenceign the node | |
| 18:47:33 | mriedem | that said you didn't provide them a tool to check when it was safe to unfence the ndoe | |
| 18:47:34 | mriedem | *node | |
| 18:47:41 | mriedem | especially for the $$$ they are giving you | |
| 18:48:16 | sean-k-mooney | true but the db recoreds showed they used --force --host to do a nova evac to the same host | |
| 18:48:24 | sean-k-mooney | which is why the evacuation failed | |
| 18:49:16 | sean-k-mooney | you cant evacuate to the same host that you are evacuating form. we should however have an api check for that | |
| 18:49:42 | sean-k-mooney | this came up last week so we havent really gotten around to figuring out what needs to be done upstream yet | |
| 18:50:00 | mriedem | well, w/o --force the scheduler would have kicked it out | |
| 18:50:06 | sean-k-mooney | yep | |
| 18:50:07 | mriedem | as long as the compute was still disabled | |
| 18:50:19 | mriedem | we should probably deprecate the --force option on the evacuate cli https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-evacuate | |
| 18:50:21 | sean-k-mooney | if the compute was active it would have kicked it out too | |
| 18:50:39 | sean-k-mooney | mriedem: didnt you already do that? | |
| 18:50:53 | mriedem | the force parameter was deprecated in the api | |
| 18:50:58 | mriedem | but it's still there on earlier microversions | |
| 18:51:28 | mriedem | added in 2.29 and removed in 267 | |
| 18:51:36 | sean-k-mooney | ah ok i remember talking about not supporting it in the osc version | |
| 18:51:49 | mloza | mriedem: I guess I hitting that orphan issue. I possibly cleared the migration records before. Now wonder I everytime I launch a new VM it goes always goes to a specific host. | |
| 18:51:49 | sean-k-mooney | speaking of which i should update that patch. | |
| 18:52:15 | mloza | What would be a remedy to this issue? | |
| 18:52:35 | sean-k-mooney | mloza: what release are you running | |
| 18:52:41 | mloza | stable/stein | |
| 18:52:57 | mriedem | sean-k-mooney: please tell me you ran pep8 on this https://review.opendev.org/#/c/669738/8.. | |
| 18:52:58 | sean-k-mooney | ok i was wondering if you had the old failed builds behavior. | |
| 18:52:59 | mriedem | https://review.opendev.org/#/c/669738/ | |
| 18:54:11 | sean-k-mooney | i think i did | |
| 18:54:18 | sean-k-mooney | i know i had to fix the odering of the imports | |
| 18:54:34 | mriedem | mloza: this is a work in progress but you might be able to use this https://review.opendev.org/#/c/670112/ - that would also provide valuable feedback on what it reports | |
| 18:55:38 | mriedem | mloza: you can use https://docs.openstack.org/osc-placement/latest/ to see what is consuming resource allocations on your nodes (resource providers) | |
| 18:55:55 | mriedem | i would start by investigating a particular host that you think should be available but placement is saying it's not | |
| 18:56:37 | mloza | will take a look | |
| 18:56:40 | mriedem | so using this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show with the --allocations option | |
| 18:57:02 | mriedem | the allocations are keyed by consumer uuid, which in the case of nova since queens can be migrations or instances | |
| 18:57:23 | mriedem | migrations for the source host during a migration, instances for the dest host | |
| 18:57:57 | mriedem | so let's say you evacuated all instances from a down host but placement is still reporting allocations against that host - those allocation consumers are likely migration records | |
| 18:57:59 | mriedem | and were orphaned | |
| 18:58:33 | mriedem | there are a couple of related bugs in this patch https://review.opendev.org/#/c/678100/ | |
| 19:18:35 | mriedem | melwitt: how are you feeling about this? https://review.opendev.org/#/c/541420/ i split the refactor out and without the compat code for old cinder it's quite a bit simpler than the last time you looked. | |
| 20:03:55 | KeithMnemonic1 | I have a customer running into an odd issue with multipath during migration (Pike). To make a long story short the LUNS assignments are getting mixed up somewhere. geguileo mentioned it possibly could be this https://review.opendev.org/#/c/551302/ I did a quick look at cherry-picking and it seems there are a ton of dependencies. Does that seems accurate or does someone have some magic to cherry-pick it to | |
| 20:03:55 | KeithMnemonic1 | pike? | |
| 20:15:29 | KeithMnemonic1 | mriedem, melwitt any thoughts on my question above | |
| 20:16:27 | mriedem | coincidentally that came up yesterday on pike https://review.opendev.org/#/c/670016/ | |
| 20:16:44 | mriedem | ^ will not fix your issue though | |
| 20:17:33 | mriedem | looking at the conflicts on the queens backport i'm not surprised that there would be conflicts going to pike, but w/o doing it myself i do'nt know how bad it is, | |
| 20:17:45 | mriedem | red hat probably doesn't care to backport that since i don't think they are supporting pike | |
| 20:23:47 | mriedem | efried_pto: are you going to be running the meeting tomorrow morning or should someone else? | |
| 20:39:26 | KeithMnemonic1 | mriedem i was using aspiers git-deps and it was a long list. I am not experienced enough to know if there are any shortcuts | |
| 20:48:30 | KeithMnemonic1 | I was looking to confirm with someone if my inital look was correct (that it is too cumbersome) or if it is not that bad | |
| 20:50:10 | mriedem | i can peek in a bit | |
| 20:51:07 | melwitt | mriedem: I dunno, haven't looked at it in a long time. will go through it today | |
| 20:56:37 | melwitt | KeithMnemonic1, mriedem: I can confirm that we haven't backported https://review.opendev.org/#/c/551302/ beyond queens downstream either, so I also don't know what to expect as far as how gnarly the backport will be without actually trying to do it | |
| 20:58:12 | KeithMnemonic1 | I took a look and it seemed gnarly but was hoping for a second opinion | |
| 21:00:13 | mriedem | is this a new failure in building docs locally? | |
| 21:00:13 | mriedem | WARNING: RSVG converter command 'rsvg-convert' cannot be run. Check the rsvg_converter_bin setting | |
| 21:00:59 | KeithMnemonic1 | melwitt a plug for aspiers this is what I used to gauge the effort https://github.com/aspiers/git-deps | |
| 21:04:31 | KeithMnemonic1 | but that pulls in every dependency and i am not experienced enough to know if any could be skipped | |
| 21:05:43 | mriedem | ah i see https://github.com/openstack/nova/commit/16b9486bf7e91bfd5dc48297cee9f54b49156c93 | |
| 21:15:19 | mriedem | we need librsvg2-bin in bindep.txt | |
| 21:19:33 | mriedem | KeithMnemonic1: do you have a paste of the dependencies? i wouldn't be surprised if a bunch of the conflicts are due to like mox removal patches | |
| 21:28:11 | mriedem | melwitt: if you have a fedora or centos system available can you confirm that librsvg2-tools provides the rsvg-convert command? | |
| 21:28:53 | mriedem | which is i think just: yum provides rsvg-convert | |
| 21:28:53 | mriedem | right? | |
| 21:29:13 | melwitt | let me check | |
| 21:31:23 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Re-propose cross-cell-resize spec for Ussuri https://review.opendev.org/683002 | |
| 21:33:15 | melwitt | mriedem: you are correct, comes from librsvg2-tools | |
| 21:33:39 | mriedem | cool, thanks. that's what cinder had in their bindep but i've only got ubuntu and that's all that zuul uses for the pdf jobs | |
| 21:33:43 | melwitt | looked on centos7.3 | |
| 21:34:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add librsvg2* to bindep https://review.opendev.org/683003 | |
| 21:34:55 | mriedem | looks like we just had a zuul restart | |
| 21:40:18 | mriedem | and it's busted, "Unable to freeze job graph: 'dict_keys' object does not support indexing" | |
| 21:40:28 | mriedem | on that note, looks like i'm out of there for the day | |
| 21:49:00 | mriedem_afk | KeithMnemonic1: quickly glancing at that pike backport from queens, there are a bunch of conflicts but the big ones look like they are due to the multiattach volume support added in queens but those aren't functional dependencies, so i don't think those would actually impact backporting the fix to pike except it makes the backport harder | |
| 22:34:46 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020 | |
| 22:34:54 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020 | |
| 22:36:19 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027 | |
| 22:36:30 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027 | |
| 22:39:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031 | |
| 22:39:58 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031 | |
| 22:41:13 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299 | |
| 22:41:19 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299 | |
| 22:42:06 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344 | |
| 22:42:18 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344 | |
| 22:43:30 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673 | |
| 22:43:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673 | |
| 22:44:50 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |