| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-05 | |||
| 12:50:39 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Re-propose expose-auto-converge-post-copy spec for Ussuri https://review.opendev.org/687199 | |
| 13:07:09 | efried | gibi_ptg: thanks for the update, and thanks to y'all for covering the session. | |
| 13:07:35 | efried | and sorry about prematurely completing that bp, forgot about the lm bits. | |
| 13:14:25 | sean-k-mooney | is the PTG starting tonight | |
| 13:17:06 | gibi_ptg | sean-k-mooney: it starting wednesday morning shanghai time | |
| 13:17:22 | gibi_ptg | efried: no problemo | |
| 13:18:21 | sean-k-mooney | gibi_ptg: ya so thats 12-16 from now right is tuesday evenig/night for you now? | |
| 13:19:07 | sean-k-mooney | ye are utc+8 right? | |
| 13:19:09 | gibi_ptg | 21:19 local time | |
| 13:19:25 | sean-k-mooney | ya that is what i guessed cool | |
| 13:50:07 | openstackgerrit | Merged openstack/nova stable/stein: Nova compute: add in log exception to help debug failures https://review.opendev.org/692936 | |
| 13:50:12 | openstackgerrit | Merged openstack/nova stable/queens: cleanup evacuated instances not on hypervisor https://review.opendev.org/687873 | |
| 13:50:17 | openstackgerrit | Merged openstack/nova stable/queens: Add functional test for resize crash compute restart revert https://review.opendev.org/687874 | |
| 13:50:22 | openstackgerrit | Merged openstack/nova stable/queens: Prevent init_host test to interfere with other tests https://review.opendev.org/687876 | |
| 13:50:28 | openstack | bug 1833581 in OpenStack Compute (nova) queens "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Elod Illes (elod-illes) | |
| 13:50:28 | openstackgerrit | Merged openstack/nova stable/queens: Functional reproduce for bug 1833581 https://review.opendev.org/687877 | |
| 13:50:35 | openstackgerrit | Merged openstack/nova stable/queens: Error out interrupted builds https://review.opendev.org/687878 | |
| 14:06:58 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Nova compute: add in log exception to help debug failures https://review.opendev.org/693009 | |
| 14:24:25 | mriedem | dansmith: here is that follow up https://review.opendev.org/#/c/692856/ - the next one after it is really simple as well | |
| 15:06:14 | artom | mriedem, left a thought on https://review.opendev.org/#/c/594139/1 | |
| 15:07:41 | watersj | How do I cancel a pre-migration task? It is stuck and will not complete | |
| 15:07:43 | watersj | ? | |
| 15:09:43 | mriedem | watersj: i think you're looking for https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort | |
| 15:12:17 | watersj | mriedem, I get instance is in an invalid state 409 | |
| 15:18:25 | efried | mriedem: not sure if we do this for storyboard bugs, but https://review.opendev.org/693028 | |
| 15:19:16 | mriedem | watersj: what is the migration status for that instance? you can use https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-server-migration-list | |
| 15:19:40 | mriedem | efried: e-r doesn't support storyboard | |
| 15:20:03 | mriedem | https://storyboard.openstack.org/#!/story/2004745 | |
| 15:20:06 | efried | oh. Oh well. The logstash query works. Is there a way to make it stick? | |
| 15:20:15 | dansmith | mriedem: WAT :P | |
| 15:20:32 | mriedem | efried: you could use https://bugs.launchpad.net/python-openstackclient/ ? | |
| 15:20:35 | mriedem | oops | |
| 15:20:37 | efried | yeah | |
| 15:20:52 | mriedem | efried: i meant https://bugs.launchpad.net/openstack-gate | |
| 15:20:56 | mriedem | that's the catch all | |
| 15:21:42 | efried | Okay. | |
| 15:21:46 | mriedem | artom: i'm looking and responding, but it's...complicated | |
| 15:22:23 | artom | mriedem, I believe you :) | |
| 15:24:08 | efried | mriedem: done, thanks. | |
| 15:28:18 | mriedem | artom: replied | |
| 15:30:00 | artom | mriedem, ack, lemme dig into that | |
| 15:30:17 | mriedem | tl;dr the neutronv2/api code is a gd mess when it comes to migrations | |
| 15:30:27 | mriedem | there are like at least 6 methods involved in mucking with port bindings | |
| 15:37:32 | artom | mriedem, I guess it comes down to "are we backporting this?" | |
| 15:37:41 | artom | If not, we can invest in a more thorough refactor/rethink | |
| 15:38:21 | mriedem | it probably depends on if sean-k-mooney can recreate the failure in a real env and confirm the bug and the fix | |
| 15:38:28 | mriedem | on...rocky | |
| 15:38:45 | mriedem | like i said in the change over a year ago, "Waiting for Sean Mooney to test this out in his environment since he was able to generate the issue and report the bug, so I want to know if this fixes it. Also, I want Miguel to take a look at this given my assumptions about how I thought neutron would handle the source host binding when the dest binding is deleted." | |
| 15:39:46 | mriedem | even if we did backport, i could do either of my 1 or 2 options and make it backportable | |
| 15:40:11 | mriedem | and add a wrapper/convenience method which doesn't get backported that calls both the activate and teardown methods under the covers | |
| 15:41:07 | sean-k-mooney | am what do i need to repoduce? reads back | |
| 15:41:17 | mriedem | https://review.opendev.org/#/c/594139/ | |
| 15:42:14 | artom | I kinda wonder what ovs-dpdk and/or hugepages have to do with it | |
| 15:42:43 | sean-k-mooney | mriedem: that sounds like https://review.opendev.org/#/c/653506/ | |
| 15:42:45 | artom | Seems like we could repro it in func tests with an instance with some ports | |
| 15:43:16 | sean-k-mooney | although i need to reload the context for both back into my head | |
| 15:43:51 | mriedem | artom: the neutron fixture would have to start tracking port bindings and do it correctly, which i'm not entirely confident in how that api works - which is why we have this bug in the first place | |
| 15:44:04 | mriedem | i.e. deleting an inactive port binding does not automatically activate the other port binding | |
| 15:44:22 | mriedem | but activating an inactive port binding automatically deactivates the previously active port binding | |
| 15:44:26 | artom | mriedem, ah, right | |
| 15:44:47 | mriedem | from my commit message, "I had assumed neutron would automatically activate the old remaining port binding when the dest host port binding was deleted, like how neutron automatically deactivates the source host binding when the dest host binding is activated, but apparently that is not the case, so we have to do it ourselves on rollback." | |
| 15:45:58 | artom | mriedem, yep - as I said in my first comment, I can't speak to the raison d'etre of the patch | |
| 15:46:00 | mriedem | so i think tl;dr on live migration rollback since rocky we leave the source host port bindings inactive and you likely need to do something to resolve that, i'm not sure that the heal instance periodic fixes it | |
| 15:46:00 | artom | Just the mechanics (assuming we need it in the first place) | |
| 15:46:14 | sean-k-mooney | yep | |
| 15:46:50 | sean-k-mooney | well we have two patach to resovle to related edgecases already | |
| 15:47:05 | sean-k-mooney | artom: what bug are you looking at | |
| 15:47:17 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 15:47:17 | artom | sean-k-mooney, https://bugs.launchpad.net/nova/+bug/1788014 | |
| 15:47:36 | watersj | mriedem, vm status is pre-migrating | |
| 15:47:40 | mriedem | https://review.opendev.org/#/c/603844/ landed in stein but i'm not sure that heals the port binding on the source host either | |
| 15:48:15 | mriedem | watersj: i wanted to know the migration record status, not the server vm_state | |
| 15:48:24 | mriedem | or maybe that's what you meant | |
| 15:48:39 | openstack | Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 15:48:39 | sean-k-mooney | so that is a duplicate of https://bugs.launchpad.net/nova/+bug/1822884 | |
| 15:48:42 | watersj | mriedem, output from server-migration-list is empty | |
| 15:49:03 | mriedem | i guess because the api doesn't consider pre-migrating as in-progress...i think i found that out recently | |
| 15:49:33 | artom | mriedem, wait, that rings a bell | |
| 15:49:35 | mriedem | anyway, https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort won't do anything for a pre-migrating migration, only running/queued/preparing | |
| 15:49:51 | mriedem | artom: yeah i noticed it when reviewing your numa live migration series | |
| 15:50:14 | sean-k-mooney | there was a seperate issue with evacuate and pre-migrationg that artom might be thinking of | |
| 15:50:16 | artom | mriedem, oh right - https://review.opendev.org/#/c/684409/ | |
| 15:50:21 | watersj | this is stein that I am running | |
| 15:50:23 | mriedem | because in the long ago live migration records didn't have pre-migrating since that was only done during the RT claim | |
| 15:50:42 | artom | But my patch only changes the non-API version of that query | |
| 15:51:24 | artom | Which is your comment on https://review.opendev.org/#/c/684409/1/nova/db/sqlalchemy/api.py@4431 | |
| 15:52:14 | mriedem | right, so the api only shows in-progress live migrations when they have status "'queued', 'preparing', 'running'" | |
| 15:52:18 | mriedem | which watersj just confirmed | |
| 15:52:37 | mriedem | even though arguably 'accepted' and 'pre-migrating' could be considered in-progress | |
| 15:52:44 | mriedem | or as in-progress as queued | |
| 15:53:07 | mriedem | watersj: did you trace down where the migration request failed and got stuck? | |
| 15:53:43 | artom | mriedem, I think that has to do with when they get put in the queue | |
| 15:53:50 | artom | IOW, you can't abort it if it's not in the queue yet | |
| 15:53:58 | mriedem | watersj: if the guest is still on the source host then nothing has really changed. you would likely have to manually change the migration record status to 'error' and reset the instance vm_state to 'active' and task_state to None | |
| 15:54:07 | watersj | no, looking at nova-compute logs did not give me much info. Currently vm is down and I trying to get it back on | |
| 15:54:35 | watersj | i did change its state ti error | |
| 15:54:39 | watersj | and then active | |
| 15:54:42 | mriedem | watersj: ok, so you'll have to set the migration record status to 'error' and hard reboot the guest i think | |
| 15:54:48 | sean-k-mooney | artom: mriedem by the way i though when we got to post-copy stage we cant rollback anymore | |
| 15:54:53 | mriedem | watersj: the instances table record or the migrations table record? | |