| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-02 | |||
| 09:17:41 | bhagyashris | gibi: I am sure I havent set force to true | |
| 09:19:26 | bhagyashris | gibi: openstack server migrate <instance-id> --live <destination-host> --block-migration this command I used for live migration | |
| 09:26:18 | gibi | bhagyashris: as you did not specify any api microverion openstack client used the minimum and in there specifying a host means force by default | |
| 09:26:21 | gibi | bhagyashris: https://developer.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action | |
| 09:26:43 | gibi | bhagyashris: this explains why you don't see anything in the nova-sch log | |
| 09:30:14 | bhagyashris | gibi: ohh ok | |
| 09:33:22 | gibi | bhagyashris: could you check what is the status of the first live migration attempt with nova migration-list command? | |
| 09:34:25 | bhagyashris | gibi: it's showing me the status as error | |
| 09:35:58 | gibi | bhagyashris: and nothing relevan on the source compute or dest compute n-cpu log? | |
| 09:38:35 | bhagyashris | gibi: at destination compute it' check destination check data is LibvirtLiveMigrateData | |
| 09:43:14 | gibi | bhagyashris: I'm clueless how that migration failed without leaving any trace of information anywhere what was the reason fo the failure. I think when it failed it failed to clean up some port binding stuff in Neutron and that causes the second migration attempt to fail at port binding | |
| 09:45:12 | bhagyashris | gibi: yeah... but at very first why it's not migrating that instance on destination node with any failure :( | |
| 09:52:34 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add microversion 2.67 to support volume_type https://review.openstack.org/606398 | |
| 10:18:27 | openstackgerrit | Sam Morrison proposed openstack/python-novaclient master: Fix up userdata argument to rebuild. https://review.openstack.org/605341 | |
| 10:32:53 | openstackgerrit | Merged openstack/nova stable/pike: Fix nova-status "_check_resource_providers" check https://review.openstack.org/600113 | |
| 10:34:15 | openstackgerrit | Merged openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947 | |
| 10:42:16 | sean-k-mooney | gibi: that livemigration error bhagyashris had is bizarr i have never seen neutron trigger FlushError i would have liked to see the n-cpu log on the source and dest node i suspect that there was a libvirt/qemu error that resulted in the first failure but not sure what would cause the second. | |
| 10:46:35 | gibi | sean-k-mooney: bhagyashris stated that there was no relevant log in the nova-cpu services :/ | |
| 10:46:54 | gibi | sean-k-mooney: but I share your feeling about the weiredness of the fault | |
| 10:49:33 | sean-k-mooney | ya i would still be interested in seeing the libvirt qemu instance log to confirm there were no qemu/libvirt errors but it is strange. im wonder if the neutron issue could have something to do with multiple port bindings. e.g. we know 1 or 2 bindings now works but did anyone ever test with 3. with out a way to reproduce however its really just guess work on my part | |
| 10:51:24 | openstackgerrit | Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436 | |
| 12:03:36 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update the contributor guide https://review.openstack.org/606926 | |
| 12:04:08 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update the CLI reference https://review.openstack.org/606871 | |
| 12:44:48 | gibi | sean-k-mooney: same here, without bhagyashris this remains a mystery | |
| 13:07:09 | mdbooth | gibi efried: I just got hit by my own patch which landed earlier: http://logs.openstack.org/36/605436/4/check/nova-tox-functional-py35/66bf4c1/job-output.txt.gz | |
| 13:07:28 | mdbooth | Just convincing myself that this is genuinely uncovering yet another race | |
| 13:09:51 | efried | mdbooth: This is good. Very good. | |
| 13:10:35 | mdbooth | efried: Ok, I can reproduce it non-deterministically locally... 'Hopefully' that means it's actually a new race rather than a bug in my other patch :) | |
| 13:10:59 | efried | s/new/latent/ <== good | |
| 13:11:17 | mdbooth | efried: Indeed, an important clarification ;) | |
| 13:14:20 | mdbooth | Ah... actually in this case it's because my patch invalidated some assumptions in the testcase | |
| 13:14:28 | mdbooth | So.. still good | |
| 13:15:02 | mdbooth | 'my patch' being my other patch | |
| 13:31:23 | openstack | bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,In progress] https://launchpad.net/bugs/1763181 - Assigned to Matthew Booth (mbooth-9) | |
| 13:31:23 | efried | mdbooth: ah, in fact https://review.openstack.org/#/c/605436/ apparently does not closes-bug #1763181 | |
| 13:31:39 | efried | but your other patch has made it manifest differently. | |
| 13:31:51 | efried | which may, please gods, make it figure-outable. | |
| 13:32:09 | mdbooth | efried: Oh, I didn't remove that? | |
| 13:32:17 | mdbooth | efried: I have a different patch which actually fixes that | |
| 13:32:46 | efried | okay, I left a reminder with a -1 :) | |
| 13:32:56 | mdbooth | On, no | |
| 13:33:16 | mdbooth | efried: No, actually it really does fix it | |
| 13:33:34 | efried | But but but | |
| 13:33:41 | efried | the same bug is showing up in the test results | |
| 13:33:44 | mdbooth | efried: That's a bug in the patch :) | |
| 13:33:51 | mdbooth | It's a different bug | |
| 13:33:57 | efried | now I'm thoroughly confused. | |
| 13:34:32 | openstack | bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,In progress] https://launchpad.net/bugs/1763181 - Assigned to Matthew Booth (mbooth-9) | |
| 13:34:32 | efried | you're saying that https://review.openstack.org/#/c/605436/ fixes bug #1763181, but there's a *different* race in parallel_evacuate? | |
| 13:34:34 | mdbooth | So the functional test used to assume that stuff was deterministically executed in order X | |
| 13:34:50 | mdbooth | My fix for the actual bug broke that assumption in the test | |
| 13:34:59 | mdbooth | I'm just fixing it | |
| 13:35:21 | mdbooth | So it does fix the issue, but it causes the same test to fail differently with a new bug that I just introduced ;) | |
| 13:35:33 | efried | Nice. So... we'll ultimately see those patches in a series together? | |
| 13:35:45 | mdbooth | No, I'll just fix the bug and resubmit in a minute. | |
| 13:36:03 | efried | in https://review.openstack.org/#/c/605436/ ? | |
| 13:36:58 | mdbooth | efried: Yep | |
| 13:37:04 | efried | okay | |
| 13:37:14 | efried | I have clarified my comment | |
| 13:37:19 | efried | fwiw | |
| 13:37:22 | mdbooth | efried: It's a bug in my new patch. | |
| 13:37:31 | mdbooth | It's a bug in my patch to fix the bug. | |
| 13:37:57 | efried | I think I understand now. I'll stop bugging you and let you fix the bug in the fix. | |
| 13:38:02 | mdbooth | efried: ;) | |
| 13:38:50 | mdbooth | efried: I only highlighted it originally, because I being super-alert to the possibility of having to revert my other patch. | |
| 13:38:55 | efried | but if I remember, I'll try to get several successful rechecks before approving. | |
| 13:39:23 | mdbooth | efried: That will take weeks at the moment ;) | |
| 13:39:34 | mdbooth | But sure | |
| 13:39:45 | efried | okay, "successful" meaning that specific test passes. | |
| 13:40:01 | efried | the runs are only taking a couple of hours. | |
| 13:54:40 | sean-k-mooney | melwitt mriedem so i finally got time to start working on sriov livemigration and i can confirm that we "broke" macvtap based sriov live migration in rocky when we added the multiple port binding. | |
| 13:55:27 | sean-k-mooney | melwitt: mriedem im not sure why yet but i think its due to how we create the dest binding so this might be a neutron error or nova error depending on the intended behavior | |
| 14:00:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: support claim_resources https://review.openstack.org/583667 | |
| 14:02:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: support claim_resources https://review.openstack.org/583667 | |
| 14:05:58 | mriedem | sean-k-mooney: super | |
| 14:07:13 | sean-k-mooney | mriedem: i think we need to set the vnic-type when creating the binding but i also was under the impression that neutorn was ment to copy that info form the source binding if not set so not sure its our bug. | |
| 14:08:39 | sean-k-mooney | mriedem: ill file a bug but it should hopefully be a trival backport. that said "sriov migration" even when it works is not reallly supprot yet | |
| 14:09:16 | sean-k-mooney | im hoping to work on that this cycle but we will see how thing go. | |
| 14:44:04 | jaypipes | sean-k-mooney: let's get down to brass tacks on https://review.openstack.org/#/c/603805/2/specs/stein/approved/device-discovery-policy-lib.rst. What is the likelihood that this will proceed? I have a number of concerns about the proposed implementation, but I don't really want to spend a bunch of time on reviewing it if you're not going to have time to pursue this and nobody else is either. | |
| 14:45:34 | efried | jaypipes: I will pursue it to whatever extent sean-k-mooney doesn't have time. Depending which approach you (and/or others) prefer, I will pursue as appropriate. | |
| 14:46:01 | jaypipes | efried: I thought you were more into pursuing Konstantinos' spec? | |
| 14:46:22 | efried | jaypipes: They're both shooting at the same target. | |
| 14:46:57 | efried | jaypipes: I like my baby better, possibly mostly because it's my baby (though I do have some logical reasons), but it's not a big gap. | |
| 14:47:42 | efried | jaypipes: So whichever approach (the spirit, not the specifics yet - as I said, neither is perfectly ready for primetime yet) you and/or others prefer, I'm going to run with. | |
| 14:48:00 | efried | But I want to run with one of them asap. | |
| 14:49:16 | efried | fwiw, I already have code written for the schema in kosamara's spec. It's in nova-powervm atm, but trivial to port. | |
| 14:50:32 | jaypipes | efried: I only have time this morning to review one of those specs. I will review kosamara's then. | |
| 14:51:19 | efried | jaypipes: ack, thanks. | |
| 14:51:54 | efried | note that a number of issues have been mentioned and acked, but not yet revised for, in previous patch sets. | |
| 14:53:11 | sean-k-mooney | jaypipes: well that depend i would really like to pursue this but my primary focus this release will be fixing more live migration stuff specificaly for sriov | |
| 14:54:06 | sean-k-mooney | jaypipes: spend time reviewing kosamara's proposal. | |
| 15:04:17 | openstackgerrit | Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436 | |
| 15:11:38 | stephenfin | johnthetubaguy: When you're about, fancy taking another look at https://review.openstack.org/#/c/456572 | |
| 15:13:56 | sean-k-mooney | mriedem: so yes i just confirmed we need to pass the vnic type to the binding api but i also found another neutron bug | |
| 15:14:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable nested allocation candidates in scheduler https://review.openstack.org/585672 | |
| 15:14:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use provider tree in virt FakeDriver https://review.openstack.org/604083 | |
| 15:15:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor allocation checking in functional tests https://review.openstack.org/607287 | |
| 15:15:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Run ServerMovingTests with nested resources https://review.openstack.org/604084 | |
| 15:15:44 | sean-k-mooney | gibi: so while macvtap sriov livemigration i was able to cause the same neutron db confilct and fush error so it looks liek tehre is definetly a neutron portbining bug at least on master | |
| 15:19:12 | gibi | sean-k-mooney: good info | |