| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-05 | |||
| 02:00:46 | mriedem | gibi_ptg: fixed | |
| 02:02:02 | gibi_ptg | mriedem: thanks | |
| 02:15:18 | mriedem | it's alive http://www.danplanet.com/blog/2019/11/04/start-and-monitor-image-pre-cache-operations-in-nova/ | |
| 02:18:04 | gibi_ptg | mriedem: I will mention it in the nova project update as something we already done from the Ussuri plans | |
| 02:19:06 | mriedem | the image cache thing? yeah good idea. | |
| 02:19:26 | gibi_ptg | yepp | |
| 02:19:54 | gibi_ptg | the image download already come up in the war stories session yesterday | |
| 02:29:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Create BDM in API during attach volume https://review.opendev.org/692940 | |
| 03:16:06 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add integration testing for heal_allocations https://review.opendev.org/692923 | |
| 03:16:08 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | |
| 03:16:08 | openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt) | |
| 03:16:08 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | |
| 05:20:56 | gibi_ptg | nova project update went OK in Shanghai. We got questions about SRIOV live migration with bonded virtio interface for failover. adrianc wil; lead a discussion around it on the PTG | |
| 08:26:20 | openstackgerrit | Tushar Patil proposed openstack/nova-specs master: Allow compute nodes to use DISK_GB from shared storage RP https://review.opendev.org/650188 | |
| 11:06:19 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | |
| 11:13:20 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | |
| 12:50:39 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Re-propose expose-auto-converge-post-copy spec for Ussuri https://review.opendev.org/687199 | |
| 13:07:09 | efried | gibi_ptg: thanks for the update, and thanks to y'all for covering the session. | |
| 13:07:35 | efried | and sorry about prematurely completing that bp, forgot about the lm bits. | |
| 13:14:25 | sean-k-mooney | is the PTG starting tonight | |
| 13:17:06 | gibi_ptg | sean-k-mooney: it starting wednesday morning shanghai time | |
| 13:17:22 | gibi_ptg | efried: no problemo | |
| 13:18:21 | sean-k-mooney | gibi_ptg: ya so thats 12-16 from now right is tuesday evenig/night for you now? | |
| 13:19:07 | sean-k-mooney | ye are utc+8 right? | |
| 13:19:09 | gibi_ptg | 21:19 local time | |
| 13:19:25 | sean-k-mooney | ya that is what i guessed cool | |
| 13:50:07 | openstackgerrit | Merged openstack/nova stable/stein: Nova compute: add in log exception to help debug failures https://review.opendev.org/692936 | |
| 13:50:12 | openstackgerrit | Merged openstack/nova stable/queens: cleanup evacuated instances not on hypervisor https://review.opendev.org/687873 | |
| 13:50:17 | openstackgerrit | Merged openstack/nova stable/queens: Add functional test for resize crash compute restart revert https://review.opendev.org/687874 | |
| 13:50:22 | openstackgerrit | Merged openstack/nova stable/queens: Prevent init_host test to interfere with other tests https://review.opendev.org/687876 | |
| 13:50:28 | openstack | bug 1833581 in OpenStack Compute (nova) queens "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Elod Illes (elod-illes) | |
| 13:50:28 | openstackgerrit | Merged openstack/nova stable/queens: Functional reproduce for bug 1833581 https://review.opendev.org/687877 | |
| 13:50:35 | openstackgerrit | Merged openstack/nova stable/queens: Error out interrupted builds https://review.opendev.org/687878 | |
| 14:06:58 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Nova compute: add in log exception to help debug failures https://review.opendev.org/693009 | |
| 14:24:25 | mriedem | dansmith: here is that follow up https://review.opendev.org/#/c/692856/ - the next one after it is really simple as well | |
| 15:06:14 | artom | mriedem, left a thought on https://review.opendev.org/#/c/594139/1 | |
| 15:07:41 | watersj | How do I cancel a pre-migration task? It is stuck and will not complete | |
| 15:07:43 | watersj | ? | |
| 15:09:43 | mriedem | watersj: i think you're looking for https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort | |
| 15:12:17 | watersj | mriedem, I get instance is in an invalid state 409 | |
| 15:18:25 | efried | mriedem: not sure if we do this for storyboard bugs, but https://review.opendev.org/693028 | |
| 15:19:16 | mriedem | watersj: what is the migration status for that instance? you can use https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-server-migration-list | |
| 15:19:40 | mriedem | efried: e-r doesn't support storyboard | |
| 15:20:03 | mriedem | https://storyboard.openstack.org/#!/story/2004745 | |
| 15:20:06 | efried | oh. Oh well. The logstash query works. Is there a way to make it stick? | |
| 15:20:15 | dansmith | mriedem: WAT :P | |
| 15:20:32 | mriedem | efried: you could use https://bugs.launchpad.net/python-openstackclient/ ? | |
| 15:20:35 | mriedem | oops | |
| 15:20:37 | efried | yeah | |
| 15:20:52 | mriedem | efried: i meant https://bugs.launchpad.net/openstack-gate | |
| 15:20:56 | mriedem | that's the catch all | |
| 15:21:42 | efried | Okay. | |
| 15:21:46 | mriedem | artom: i'm looking and responding, but it's...complicated | |
| 15:22:23 | artom | mriedem, I believe you :) | |
| 15:24:08 | efried | mriedem: done, thanks. | |
| 15:28:18 | mriedem | artom: replied | |
| 15:30:00 | artom | mriedem, ack, lemme dig into that | |
| 15:30:17 | mriedem | tl;dr the neutronv2/api code is a gd mess when it comes to migrations | |
| 15:30:27 | mriedem | there are like at least 6 methods involved in mucking with port bindings | |
| 15:37:32 | artom | mriedem, I guess it comes down to "are we backporting this?" | |
| 15:37:41 | artom | If not, we can invest in a more thorough refactor/rethink | |
| 15:38:21 | mriedem | it probably depends on if sean-k-mooney can recreate the failure in a real env and confirm the bug and the fix | |
| 15:38:28 | mriedem | on...rocky | |
| 15:38:45 | mriedem | like i said in the change over a year ago, "Waiting for Sean Mooney to test this out in his environment since he was able to generate the issue and report the bug, so I want to know if this fixes it. Also, I want Miguel to take a look at this given my assumptions about how I thought neutron would handle the source host binding when the dest binding is deleted." | |
| 15:39:46 | mriedem | even if we did backport, i could do either of my 1 or 2 options and make it backportable | |
| 15:40:11 | mriedem | and add a wrapper/convenience method which doesn't get backported that calls both the activate and teardown methods under the covers | |
| 15:41:07 | sean-k-mooney | am what do i need to repoduce? reads back | |
| 15:41:17 | mriedem | https://review.opendev.org/#/c/594139/ | |
| 15:42:14 | artom | I kinda wonder what ovs-dpdk and/or hugepages have to do with it | |
| 15:42:43 | sean-k-mooney | mriedem: that sounds like https://review.opendev.org/#/c/653506/ | |
| 15:42:45 | artom | Seems like we could repro it in func tests with an instance with some ports | |
| 15:43:16 | sean-k-mooney | although i need to reload the context for both back into my head | |
| 15:43:51 | mriedem | artom: the neutron fixture would have to start tracking port bindings and do it correctly, which i'm not entirely confident in how that api works - which is why we have this bug in the first place | |
| 15:44:04 | mriedem | i.e. deleting an inactive port binding does not automatically activate the other port binding | |
| 15:44:22 | mriedem | but activating an inactive port binding automatically deactivates the previously active port binding | |
| 15:44:26 | artom | mriedem, ah, right | |
| 15:44:47 | mriedem | from my commit message, "I had assumed neutron would automatically activate the old remaining port binding when the dest host port binding was deleted, like how neutron automatically deactivates the source host binding when the dest host binding is activated, but apparently that is not the case, so we have to do it ourselves on rollback." | |
| 15:45:58 | artom | mriedem, yep - as I said in my first comment, I can't speak to the raison d'etre of the patch | |
| 15:46:00 | mriedem | so i think tl;dr on live migration rollback since rocky we leave the source host port bindings inactive and you likely need to do something to resolve that, i'm not sure that the heal instance periodic fixes it | |
| 15:46:00 | artom | Just the mechanics (assuming we need it in the first place) | |
| 15:46:14 | sean-k-mooney | yep | |
| 15:46:50 | sean-k-mooney | well we have two patach to resovle to related edgecases already | |
| 15:47:05 | sean-k-mooney | artom: what bug are you looking at | |
| 15:47:17 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 15:47:17 | artom | sean-k-mooney, https://bugs.launchpad.net/nova/+bug/1788014 | |
| 15:47:36 | watersj | mriedem, vm status is pre-migrating | |
| 15:47:40 | mriedem | https://review.opendev.org/#/c/603844/ landed in stein but i'm not sure that heals the port binding on the source host either | |
| 15:48:15 | mriedem | watersj: i wanted to know the migration record status, not the server vm_state | |
| 15:48:24 | mriedem | or maybe that's what you meant | |
| 15:48:39 | openstack | Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 15:48:39 | sean-k-mooney | so that is a duplicate of https://bugs.launchpad.net/nova/+bug/1822884 | |
| 15:48:42 | watersj | mriedem, output from server-migration-list is empty | |
| 15:49:03 | mriedem | i guess because the api doesn't consider pre-migrating as in-progress...i think i found that out recently | |
| 15:49:33 | artom | mriedem, wait, that rings a bell | |
| 15:49:35 | mriedem | anyway, https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort won't do anything for a pre-migrating migration, only running/queued/preparing | |
| 15:49:51 | mriedem | artom: yeah i noticed it when reviewing your numa live migration series | |
| 15:50:14 | sean-k-mooney | there was a seperate issue with evacuate and pre-migrationg that artom might be thinking of | |
| 15:50:16 | artom | mriedem, oh right - https://review.opendev.org/#/c/684409/ | |
| 15:50:21 | watersj | this is stein that I am running | |
| 15:50:23 | mriedem | because in the long ago live migration records didn't have pre-migrating since that was only done during the RT claim | |