Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-02
08:54:42 bhagyashris gibi: this is log I get at neutron server side http://paste.openstack.org/show/731235/
08:55:16 bhagyashris gibi: yes I have specified target host in request but not force=True
08:57:08 openstackgerrit Merged openstack/nova master: Fix neutron-tempest-linuxbridge irrelevant-files https://review.openstack.org/606616
08:57:17 openstackgerrit Merged openstack/nova master: Add hide server address tests in test_serversV21.py https://review.openstack.org/605435
08:57:27 openstackgerrit Merged openstack/nova stable/ocata: [Stable Only] Remove soft-deleted instances from quota_usages https://review.openstack.org/570653
08:57:38 openstackgerrit Merged openstack/nova master: Follow up for Ib6f95c22ffd3ea235b60db4da32094d49c2efa2a https://review.openstack.org/604743
08:57:40 gibi bhagyashris: if force is False then nova checks the host with the scheduler. did you looked at the scheduler log for the first migration attempt?
08:57:45 openstackgerrit Merged openstack/nova master: Raise error on timeout in wait_for_versioned_notifications https://review.openstack.org/604859
09:01:24 bhagyashris gibi: checked the n-sch log but there I didn't get any logs
09:01:53 gibi bhagyashris: it is soo strange that nothing in the nova-conductor log
09:02:27 bhagyashris gibi: n-cond or n-sch?
09:02:34 gibi nova-cond
09:02:40 bhagyashris At n-cond I got the logs
09:02:59 gibi bhagyashris: could you post the n-cond log?
09:06:35 bhagyashris gibi: #link: http://paste.openstack.org/show/731237/
09:16:49 gibi bhagyashris: are you sure that force wasn't set to True in the live migration request? the last log line in http://paste.openstack.org/show/731237/ shows that claim_resources_on_destination was called to make the allocation for the instance on the dest host but that function only called for force live migrate and force evacuate
09:17:41 bhagyashris gibi: I am sure I havent set force to true
09:19:26 bhagyashris gibi: openstack server migrate <instance-id> --live <destination-host> --block-migration this command I used for live migration
09:26:18 gibi bhagyashris: as you did not specify any api microverion openstack client used the minimum and in there specifying a host means force by default
09:26:21 gibi bhagyashris: https://developer.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action
09:26:43 gibi bhagyashris: this explains why you don't see anything in the nova-sch log
09:30:14 bhagyashris gibi: ohh ok
09:33:22 gibi bhagyashris: could you check what is the status of the first live migration attempt with nova migration-list command?
09:34:25 bhagyashris gibi: it's showing me the status as error
09:35:58 gibi bhagyashris: and nothing relevan on the source compute or dest compute n-cpu log?
09:38:35 bhagyashris gibi: at destination compute it' check destination check data is LibvirtLiveMigrateData
09:43:14 gibi bhagyashris: I'm clueless how that migration failed without leaving any trace of information anywhere what was the reason fo the failure. I think when it failed it failed to clean up some port binding stuff in Neutron and that causes the second migration attempt to fail at port binding
09:45:12 bhagyashris gibi: yeah... but at very first why it's not migrating that instance on destination node with any failure :(
09:52:34 openstackgerrit Brin Zhang proposed openstack/nova master: Add microversion 2.67 to support volume_type https://review.openstack.org/606398
10:18:27 openstackgerrit Sam Morrison proposed openstack/python-novaclient master: Fix up userdata argument to rebuild. https://review.openstack.org/605341
10:32:53 openstackgerrit Merged openstack/nova stable/pike: Fix nova-status "_check_resource_providers" check https://review.openstack.org/600113
10:34:15 openstackgerrit Merged openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947
10:42:16 sean-k-mooney gibi: that livemigration error bhagyashris had is bizarr i have never seen neutron trigger FlushError i would have liked to see the n-cpu log on the source and dest node i suspect that there was a libvirt/qemu error that resulted in the first failure but not sure what would cause the second.
10:46:35 gibi sean-k-mooney: bhagyashris stated that there was no relevant log in the nova-cpu services :/
10:46:54 gibi sean-k-mooney: but I share your feeling about the weiredness of the fault
10:49:33 sean-k-mooney ya i would still be interested in seeing the libvirt qemu instance log to confirm there were no qemu/libvirt errors but it is strange. im wonder if the neutron issue could have something to do with multiple port bindings. e.g. we know 1 or 2 bindings now works but did anyone ever test with 3. with out a way to reproduce however its really just guess work on my part
10:51:24 openstackgerrit Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436
12:03:36 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Update the contributor guide https://review.openstack.org/606926
12:04:08 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Update the CLI reference https://review.openstack.org/606871
12:44:48 gibi sean-k-mooney: same here, without bhagyashris this remains a mystery
13:07:09 mdbooth gibi efried: I just got hit by my own patch which landed earlier: http://logs.openstack.org/36/605436/4/check/nova-tox-functional-py35/66bf4c1/job-output.txt.gz
13:07:28 mdbooth Just convincing myself that this is genuinely uncovering yet another race
13:09:51 efried mdbooth: This is good. Very good.
13:10:35 mdbooth efried: Ok, I can reproduce it non-deterministically locally... 'Hopefully' that means it's actually a new race rather than a bug in my other patch :)
13:10:59 efried s/new/latent/ <== good
13:11:17 mdbooth efried: Indeed, an important clarification ;)
13:14:20 mdbooth Ah... actually in this case it's because my patch invalidated some assumptions in the testcase
13:14:28 mdbooth So.. still good
13:15:02 mdbooth 'my patch' being my other patch
13:31:23 openstack bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,In progress] https://launchpad.net/bugs/1763181 - Assigned to Matthew Booth (mbooth-9)
13:31:23 efried mdbooth: ah, in fact https://review.openstack.org/#/c/605436/ apparently does not closes-bug #1763181
13:31:39 efried but your other patch has made it manifest differently.
13:31:51 efried which may, please gods, make it figure-outable.
13:32:09 mdbooth efried: Oh, I didn't remove that?
13:32:17 mdbooth efried: I have a different patch which actually fixes that
13:32:46 efried okay, I left a reminder with a -1 :)
13:32:56 mdbooth On, no
13:33:16 mdbooth efried: No, actually it really does fix it
13:33:34 efried But but but
13:33:41 efried the same bug is showing up in the test results
13:33:44 mdbooth efried: That's a bug in the patch :)
13:33:51 mdbooth It's a different bug
13:33:57 efried now I'm thoroughly confused.
13:34:32 openstack bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,In progress] https://launchpad.net/bugs/1763181 - Assigned to Matthew Booth (mbooth-9)
13:34:32 efried you're saying that https://review.openstack.org/#/c/605436/ fixes bug #1763181, but there's a *different* race in parallel_evacuate?
13:34:34 mdbooth So the functional test used to assume that stuff was deterministically executed in order X
13:34:50 mdbooth My fix for the actual bug broke that assumption in the test
13:34:59 mdbooth I'm just fixing it
13:35:21 mdbooth So it does fix the issue, but it causes the same test to fail differently with a new bug that I just introduced ;)
13:35:33 efried Nice. So... we'll ultimately see those patches in a series together?
13:35:45 mdbooth No, I'll just fix the bug and resubmit in a minute.
13:36:03 efried in https://review.openstack.org/#/c/605436/ ?
13:36:58 mdbooth efried: Yep
13:37:04 efried okay
13:37:14 efried I have clarified my comment
13:37:19 efried fwiw
13:37:22 mdbooth efried: It's a bug in my new patch.
13:37:31 mdbooth It's a bug in my patch to fix the bug.
13:37:57 efried I think I understand now. I'll stop bugging you and let you fix the bug in the fix.
13:38:02 mdbooth efried: ;)
13:38:50 mdbooth efried: I only highlighted it originally, because I being super-alert to the possibility of having to revert my other patch.
13:38:55 efried but if I remember, I'll try to get several successful rechecks before approving.
13:39:23 mdbooth efried: That will take weeks at the moment ;)
13:39:34 mdbooth But sure
13:39:45 efried okay, "successful" meaning that specific test passes.
13:40:01 efried the runs are only taking a couple of hours.
13:54:40 sean-k-mooney melwitt mriedem so i finally got time to start working on sriov livemigration and i can confirm that we "broke" macvtap based sriov live migration in rocky when we added the multiple port binding.
13:55:27 sean-k-mooney melwitt: mriedem im not sure why yet but i think its due to how we create the dest binding so this might be a neutron error or nova error depending on the intended behavior
14:00:32 openstackgerrit Balazs Gibizer proposed openstack/nova master: consumer gen: support claim_resources https://review.openstack.org/583667
14:02:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: consumer gen: support claim_resources https://review.openstack.org/583667
14:05:58 mriedem sean-k-mooney: super
14:07:13 sean-k-mooney mriedem: i think we need to set the vnic-type when creating the binding but i also was under the impression that neutorn was ment to copy that info form the source binding if not set so not sure its our bug.
14:08:39 sean-k-mooney mriedem: ill file a bug but it should hopefully be a trival backport. that said "sriov migration" even when it works is not reallly supprot yet
14:09:16 sean-k-mooney im hoping to work on that this cycle but we will see how thing go.
14:44:04 jaypipes sean-k-mooney: let's get down to brass tacks on https://review.openstack.org/#/c/603805/2/specs/stein/approved/device-discovery-policy-lib.rst. What is the likelihood that this will proceed? I have a number of concerns about the proposed implementation, but I don't really want to spend a bunch of time on reviewing it if you're not going to have time to pursue this and nobody else is either.
14:45:34 efried jaypipes: I will pursue it to whatever extent sean-k-mooney doesn't have time. Depending which approach you (and/or others) prefer, I will pursue as appropriate.
14:46:01 jaypipes efried: I thought you were more into pursuing Konstantinos' spec?
14:46:22 efried jaypipes: They're both shooting at the same target.
14:46:57 efried jaypipes: I like my baby better, possibly mostly because it's my baby (though I do have some logical reasons), but it's not a big gap.
14:47:42 efried jaypipes: So whichever approach (the spirit, not the specifics yet - as I said, neither is perfectly ready for primetime yet) you and/or others prefer, I'm going to run with.

Earlier   Later