| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-01 | |||
| 22:40:20 | mriedem | so default to workaround enabled and people can disable that when they have drained their existing tokens in the consoleauth service? | |
| 22:40:31 | mriedem | wonder if there is a nova-status check we can add for this... | |
| 22:41:17 | melwitt | I was thinking not to change the default, because only people with long TTL will potentially care about avoiding an instant reset of consoles | |
| 22:41:35 | melwitt | and the workaround is for those peeps | |
| 22:41:46 | melwitt | default TTL is 10 min | |
| 22:47:21 | sean-k-mooney | i think 3 hours of debuging the neutron sriov nic agent is enough for tonight | |
| 23:00:24 | sean-k-mooney | ... so i decided to restack and the thing i was debugging for the last 3 hours is now working and the only codecages i have done is add logging | |
| 23:48:52 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Use nova-consoleauth only if workaround enabled https://review.openstack.org/607068 | |
| 23:49:41 | mriedem | melwitt: dansmith: thanks for hitting those pike backports in https://etherpad.openstack.org/p/nova-ocata-em | |
| 23:50:38 | melwitt | thanks for etherpadding them | |
| 23:50:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | |
| #openstack-nova - 2018-10-02 | |||
| 00:04:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle missing marker during online data migration https://review.openstack.org/605164 | |
| 00:05:01 | melwitt | takashin: hi, could you please check the reply to your -1 on this change? https://review.openstack.org/582147 | |
| 00:05:38 | takashin | melwitt: I will check it. | |
| 00:05:45 | melwitt | thank you | |
| 00:07:12 | takashin | melwitt: I would like to know other cores' opinions about https://review.openstack.org/#/c/582147/ . | |
| 00:07:46 | melwitt | takashin: ack. will try to look later | |
| 00:14:18 | openstackgerrit | Merged openstack/nova stable/pike: unquiesce instance after quiesce failure https://review.openstack.org/581454 | |
| 00:53:38 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947 | |
| 01:34:25 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add compute version 36 to support ``volume_type`` https://review.openstack.org/579360 | |
| 02:29:39 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove more code related to extensions and testing https://review.openstack.org/607088 | |
| 02:32:26 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove more code related to extensions and testing https://review.openstack.org/607088 | |
| 02:33:42 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove more code related to extensions and testing https://review.openstack.org/607088 | |
| 02:51:10 | openstackgerrit | Merged openstack/nova stable/pike: Update RequestSpec.flavor on resize_revert https://review.openstack.org/605879 | |
| 03:12:45 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: [WIP]Remove extensions loading framework from wsgi.py https://review.openstack.org/607092 | |
| 03:21:54 | openstackgerrit | Merged openstack/nova stable/rocky: nova-manage - fix online_data_migrations counts https://review.openstack.org/605828 | |
| 03:22:01 | openstackgerrit | Merged openstack/nova stable/pike: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604449 | |
| 03:24:16 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Merge image_size extension response into image view builder https://review.openstack.org/606845 | |
| 03:49:07 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947 | |
| 04:08:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add compute API version for when a ``volume_type`` is requested https://review.openstack.org/605573 | |
| 04:24:54 | openstackgerrit | Merged openstack/nova stable/pike: Use ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa in tree https://review.openstack.org/581444 | |
| 06:22:03 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260 | |
| 07:11:42 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: Live migration of instances https://review.openstack.org/270116 | |
| 07:54:40 | bauzas | good morning nova | |
| 08:05:01 | gryf | morning guys | |
| 08:11:03 | bhagyashris | Hi all, I am facing one issue on current master while live migration of the instance , I am trying to live migrate the instance but it's not migrating that instance on given destination node and that instance remains on source node with the status as active, checked n-cpu, n-condunctor logs but didn't get any error at first time. | |
| 08:12:06 | bhagyashris | And if I again try to live migrate that instance then it's giveing me Migration pre-check error: Binding failed for port <port-id> please check neutron logs for more information. (HTTP 400 | |
| 08:12:58 | bhagyashris | Can anyone please help me | |
| 08:39:44 | takashin | Nova cores, could you review https://review.openstack.org/#/c/606947/ ? It is a fix for a gate failure in python-novaclient. | |
| 08:48:40 | gibi | bhagyashris: what do you see in the neutron log about the bind failure? | |
| 08:52:52 | gibi | bhagyashris: have you specified target host for the migration ? Have you use force=True in the migration request? | |
| 08:54:42 | bhagyashris | gibi: this is log I get at neutron server side http://paste.openstack.org/show/731235/ | |
| 08:55:16 | bhagyashris | gibi: yes I have specified target host in request but not force=True | |
| 08:57:08 | openstackgerrit | Merged openstack/nova master: Fix neutron-tempest-linuxbridge irrelevant-files https://review.openstack.org/606616 | |
| 08:57:17 | openstackgerrit | Merged openstack/nova master: Add hide server address tests in test_serversV21.py https://review.openstack.org/605435 | |
| 08:57:27 | openstackgerrit | Merged openstack/nova stable/ocata: [Stable Only] Remove soft-deleted instances from quota_usages https://review.openstack.org/570653 | |
| 08:57:38 | openstackgerrit | Merged openstack/nova master: Follow up for Ib6f95c22ffd3ea235b60db4da32094d49c2efa2a https://review.openstack.org/604743 | |
| 08:57:40 | gibi | bhagyashris: if force is False then nova checks the host with the scheduler. did you looked at the scheduler log for the first migration attempt? | |
| 08:57:45 | openstackgerrit | Merged openstack/nova master: Raise error on timeout in wait_for_versioned_notifications https://review.openstack.org/604859 | |
| 09:01:24 | bhagyashris | gibi: checked the n-sch log but there I didn't get any logs | |
| 09:01:53 | gibi | bhagyashris: it is soo strange that nothing in the nova-conductor log | |
| 09:02:27 | bhagyashris | gibi: n-cond or n-sch? | |
| 09:02:34 | gibi | nova-cond | |
| 09:02:40 | bhagyashris | At n-cond I got the logs | |
| 09:02:59 | gibi | bhagyashris: could you post the n-cond log? | |
| 09:06:35 | bhagyashris | gibi: #link: http://paste.openstack.org/show/731237/ | |
| 09:16:49 | gibi | bhagyashris: are you sure that force wasn't set to True in the live migration request? the last log line in http://paste.openstack.org/show/731237/ shows that claim_resources_on_destination was called to make the allocation for the instance on the dest host but that function only called for force live migrate and force evacuate | |
| 09:17:41 | bhagyashris | gibi: I am sure I havent set force to true | |
| 09:19:26 | bhagyashris | gibi: openstack server migrate <instance-id> --live <destination-host> --block-migration this command I used for live migration | |
| 09:26:18 | gibi | bhagyashris: as you did not specify any api microverion openstack client used the minimum and in there specifying a host means force by default | |
| 09:26:21 | gibi | bhagyashris: https://developer.openstack.org/api-ref/compute/#live-migrate-server-os-migratelive-action | |
| 09:26:43 | gibi | bhagyashris: this explains why you don't see anything in the nova-sch log | |
| 09:30:14 | bhagyashris | gibi: ohh ok | |
| 09:33:22 | gibi | bhagyashris: could you check what is the status of the first live migration attempt with nova migration-list command? | |
| 09:34:25 | bhagyashris | gibi: it's showing me the status as error | |
| 09:35:58 | gibi | bhagyashris: and nothing relevan on the source compute or dest compute n-cpu log? | |
| 09:38:35 | bhagyashris | gibi: at destination compute it' check destination check data is LibvirtLiveMigrateData | |
| 09:43:14 | gibi | bhagyashris: I'm clueless how that migration failed without leaving any trace of information anywhere what was the reason fo the failure. I think when it failed it failed to clean up some port binding stuff in Neutron and that causes the second migration attempt to fail at port binding | |
| 09:45:12 | bhagyashris | gibi: yeah... but at very first why it's not migrating that instance on destination node with any failure :( | |
| 09:52:34 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add microversion 2.67 to support volume_type https://review.openstack.org/606398 | |
| 10:18:27 | openstackgerrit | Sam Morrison proposed openstack/python-novaclient master: Fix up userdata argument to rebuild. https://review.openstack.org/605341 | |
| 10:32:53 | openstackgerrit | Merged openstack/nova stable/pike: Fix nova-status "_check_resource_providers" check https://review.openstack.org/600113 | |
| 10:34:15 | openstackgerrit | Merged openstack/python-novaclient master: Fix test_instance_action functional test failure https://review.openstack.org/606947 | |
| 10:42:16 | sean-k-mooney | gibi: that livemigration error bhagyashris had is bizarr i have never seen neutron trigger FlushError i would have liked to see the n-cpu log on the source and dest node i suspect that there was a libvirt/qemu error that resulted in the first failure but not sure what would cause the second. | |
| 10:46:35 | gibi | sean-k-mooney: bhagyashris stated that there was no relevant log in the nova-cpu services :/ | |
| 10:46:54 | gibi | sean-k-mooney: but I share your feeling about the weiredness of the fault | |
| 10:49:33 | sean-k-mooney | ya i would still be interested in seeing the libvirt qemu instance log to confirm there were no qemu/libvirt errors but it is strange. im wonder if the neutron issue could have something to do with multiple port bindings. e.g. we know 1 or 2 bindings now works but did anyone ever test with 3. with out a way to reproduce however its really just guess work on my part | |
| 10:51:24 | openstackgerrit | Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436 | |
| 12:03:36 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update the contributor guide https://review.openstack.org/606926 | |
| 12:04:08 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Update the CLI reference https://review.openstack.org/606871 | |
| 12:44:48 | gibi | sean-k-mooney: same here, without bhagyashris this remains a mystery | |
| 13:07:09 | mdbooth | gibi efried: I just got hit by my own patch which landed earlier: http://logs.openstack.org/36/605436/4/check/nova-tox-functional-py35/66bf4c1/job-output.txt.gz | |
| 13:07:28 | mdbooth | Just convincing myself that this is genuinely uncovering yet another race | |
| 13:09:51 | efried | mdbooth: This is good. Very good. | |
| 13:10:35 | mdbooth | efried: Ok, I can reproduce it non-deterministically locally... 'Hopefully' that means it's actually a new race rather than a bug in my other patch :) | |
| 13:10:59 | efried | s/new/latent/ <== good | |
| 13:11:17 | mdbooth | efried: Indeed, an important clarification ;) | |
| 13:14:20 | mdbooth | Ah... actually in this case it's because my patch invalidated some assumptions in the testcase | |
| 13:14:28 | mdbooth | So.. still good | |
| 13:15:02 | mdbooth | 'my patch' being my other patch | |
| 13:31:23 | efried | mdbooth: ah, in fact https://review.openstack.org/#/c/605436/ apparently does not closes-bug #1763181 | |
| 13:31:23 | openstack | bug 1763181 in OpenStack Compute (nova) "test_parallel_evacuate_with_server_group intermittently fails" [Medium,In progress] https://launchpad.net/bugs/1763181 - Assigned to Matthew Booth (mbooth-9) | |
| 13:31:39 | efried | but your other patch has made it manifest differently. | |
| 13:31:51 | efried | which may, please gods, make it figure-outable. | |
| 13:32:09 | mdbooth | efried: Oh, I didn't remove that? | |
| 13:32:17 | mdbooth | efried: I have a different patch which actually fixes that | |
| 13:32:46 | efried | okay, I left a reminder with a -1 :) | |
| 13:32:56 | mdbooth | On, no | |
| 13:33:16 | mdbooth | efried: No, actually it really does fix it | |
| 13:33:34 | efried | But but but | |