| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-05 | |||
| 12:29:49 | wy | ok | |
| 12:29:54 | wy | please wait | |
| 12:35:06 | wy | INFO nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Data remaining 63950848 bytes, low watermark 729088 bytes 143 seconds ago | |
| 12:35:07 | wy | 2017-06-05 20:34:23.970 32069 ERROR nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Live Migration failure: operation aborted: migration job: canceled by client | |
| 12:35:07 | wy | 2017-06-05 20:34:23.664 32069 WARNING nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Live migration stuck for 150 sec | |
| 12:35:10 | wy | 2017-06-05 20:34:24.170 32069 ERROR nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Migration operation has aborted | |
| 12:35:14 | wy | 2017-06-05 20:34:40.872 32069 INFO nova.compute.resource_tracker [req-36ecb5cc-4060-4006-8ed4-5c196020a583 - - - - -] Auditing locally available compute resources for node compute1 | |
| 12:35:17 | wy | could this help?? | |
| 12:35:59 | wy | this has error.But I do not do any operation | |
| 12:36:22 | wy | @<kwathore> | |
| 12:36:30 | wy | @kwathore | |
| 12:36:35 | wy | Thanks | |
| 12:43:04 | wy | please help me! | |
| 12:43:12 | wy | Thanks ALL | |
| 12:45:41 | openstackgerrit | Eric Fried proposed openstack/nova master: Add service_token for nova-glance interaction https://review.openstack.org/419662 | |
| 12:47:30 | wy | sorry...i do not understand | |
| 12:48:00 | asettle | wy: that was a automated bot | |
| 12:49:39 | wy | but wy? | |
| 12:49:49 | wy | but why? | |
| 12:49:59 | wy | how can i solve the problem? | |
| 12:50:31 | asettle | wy: I don't know, have you tried googling your issue? Have you asked in ask.openstack.org? Have you asked in #openstack or #openstack-101 ? | |
| 12:51:31 | wy | google has no the problem | |
| 12:52:04 | asettle | Maybe you should try one of the other avenues | |
| 12:53:12 | wy | I am trying | |
| 12:53:23 | wy | but have no result | |
| 12:53:41 | wy | asette | |
| 12:53:51 | wy | asettle thanks a lot | |
| 12:54:36 | asettle | wy: just wait, everyone's working across the world and are not all awake at the same time. I would recommend waiting at least 48 hours for responses. If you are seriously struggling, email the openstack list with your query. | |
| 12:56:08 | wy | asettle Ok,thanks very much.I am trying other ways | |
| 12:57:49 | openstackgerrit | Eric Fried proposed openstack/nova master: remove ugly local import https://review.openstack.org/450787 | |
| 13:13:15 | openstackgerrit | Merged openstack/nova-specs master: Fix citation references (Sphinx 1.6.x compatibility) https://review.openstack.org/470252 | |
| 13:25:56 | openstack | Launchpad bug 1695634 in OpenStack Compute (nova) "Something weird just broke in creating servers" [Undecided,New] | |
| 13:25:56 | mordred | mriedem: morning sir! over the weekend I did a bunch of looking in to new gate failures we're seeing creating servers in shade's functional tests, and I wrote it down in https://bugs.launchpad.net/nova/+bug/1695634 | |
| 13:26:44 | mordred | mriedem: tl;dr - seeing new racey tests that seem to be related to compute nodes not yet being registered with their cell | |
| 13:29:37 | mriedem | mordred: comment 1 and 2 have the same link | |
| 13:30:07 | mriedem | mordred: also you might have been seeing this red herring in the scheduler logs on startup https://review.openstack.org/#/c/454426/ | |
| 13:31:12 | mriedem | now i see in later comments you reverted that one - but i don't think that's your issue | |
| 13:31:34 | mordred | mriedem: whoops: http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-sch.txt.gz#_Jun_02_12_45_15_755877 should be the second link | |
| 13:31:50 | mordred | mriedem: and yes - I agree, I'm not sure that's the issue either | |
| 13:32:32 | mriedem | because all that should do is avoid a nasty stacktrace | |
| 13:32:37 | mriedem | when we didn't really have anything to do yet anyway | |
| 13:33:19 | mordred | nod. avoiding stack-trace == good | |
| 13:35:11 | mordred | mriedem: if you look at the first link - it looks like maybe that attempt to create a server is happening before "Received a sync request from an unknown host 'ubuntu-xenial-osic-cloud1-s3500-9104911'. Re-created its InstanceList" | |
| 13:35:27 | mordred | which is, I'm guessing, when the compute host gets registered with the cell, yeah? | |
| 13:36:05 | mriedem | that's a periodic task in the compute | |
| 13:36:06 | cdent | mriedem: do you know when johnthetubaguy is expected to be back in circulation? | |
| 13:36:12 | mriedem | cdent: this week i thought | |
| 13:36:36 | mriedem | mordred: the compute service has a periodic task to send a list of it's instances to the scheduler | |
| 13:37:06 | mriedem | maybe there is something i messed up with "registering" the host on the initial startup | |
| 13:37:26 | mriedem | given this could all be disabled based on the config options you're using, i'd hope that's not the case | |
| 13:37:28 | mriedem | b/c it'd be dumb | |
| 13:37:37 | mordred | yah - because it seems like the first time it happens at http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-sch.txt.gz#_Jun_02_12_47_14_428265 ... | |
| 13:38:07 | mordred | it indicates the host is unknonw - but then that issuedoesn't happen again, so it seems that the host is registered at that point? | |
| 13:38:25 | mriedem | it registers it when it hits that sync | |
| 13:38:34 | mriedem | and just defaults to an empty instance list for that host | |
| 13:38:35 | mordred | yah | |
| 13:38:37 | mriedem | in an internal mapping | |
| 13:38:43 | mriedem | so shouldn't be a problem | |
| 13:38:47 | mordred | but I manage to try to create a server before registration | |
| 13:39:16 | mriedem | mordred: this is a single node job right? | |
| 13:39:19 | mordred | it is | |
| 13:39:28 | mordred | and it's just using devstack defaults for nova | |
| 13:41:12 | wy | morning every one! | |
| 13:41:13 | wy | ~ | |
| 13:41:14 | mriedem | i thought we used to call discover_hosts or simple_cell_setup in here https://github.com/openstack-dev/devstack/blob/master/lib/nova | |
| 13:41:18 | mriedem | but i'm not seeing that now | |
| 13:41:25 | wy | I do the live-migration successfully.However,when the VM(centos7) is playing video,the migration is failed.But,if I shut down the video,the migration is successful. | |
| 13:41:50 | wy | does the video affect the migration? | |
| 13:42:37 | wy | 2017-06-05 20:34:23.970 32069 ERROR nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Live Migration failure: operation aborted: migration job: canceled by client | |
| 13:42:37 | wy | 2017-06-05 20:34:23.664 32069 WARNING nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Live migration stuck for 150 sec | |
| 13:42:37 | wy | INFO nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Data remaining 63950848 bytes, low watermark 729088 bytes 143 seconds ago | |
| 13:42:40 | wy | 2017-06-05 20:34:24.170 32069 ERROR nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Migration operation has aborted | |
| 13:42:43 | wy | 2017-06-05 20:34:40.872 32069 INFO nova.compute.resource_tracker [req-36ecb5cc-4060-4006-8ed4-5c196020a583 - - - - -] Auditing locally available compute resources for node compute1 | |
| 13:43:19 | mordred | wy: hi! when you have things like that to share, please use http://paste.openstack.org/ and past a link instead | |
| 13:44:11 | mordred | mriedem: discover_hosts seems to be top level now | |
| 13:45:14 | edleafe | Scheduler subteam meeting in 15 minutes in #openstack-meeting-alt | |
| 13:45:39 | mordred | mriedem: http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/devstacklog.txt.gz#_2017-06-02_12_46_16_568 | |
| 13:46:33 | mriedem | 2017-06-02 12:46:19.540 | Creating host mapping for compute host 'ubuntu-xenial-osic-cloud1-s3500-9104911': bc5499e9-2e12-4ae6-9e5b-58b8104de163 | |
| 13:46:44 | mriedem | mordred: that's what we're waiting on - does your failed test run before that? | |
| 13:47:00 | mriedem | looks like the test runs after | |
| 13:47:19 | mriedem | Jun 02 12:46:58.103925 ubuntu-xenial-osic-cloud1-s3500-9104911 nova-api[16069]: INFO nova.osapi_compute.wsgi.server [None req-a92036cd-3f52-41d6-b104-cb446cfa0ca1 demo demo] 10.39.244.211 "POST /v2.1/servers HTTP/1.1" status: 202 len: 870 time: 0.6169720 | |
| 13:47:44 | wy | ok..thankd | |
| 13:49:17 | mriedem | and we blow up here http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-cond.txt.gz#_Jun_02_12_46_59_071485 | |
| 13:49:24 | wy | http://paste.openstack.org/ | |
| 13:49:53 | efried | mriedem mordred Haven't looked at the bug itself, but it sounds like it could be related to something we hit in the PowerVM CI, where discover_hosts was running before the compute service was ready to handle it. esberglu added a --strict option to discover_hosts that makes it exit nonzero unless it actually registered your host, so you can run it in a loop until it succeeds. | |
| 13:50:42 | wy | http://paste.openstack.org/show/X5rwWcSIkukpRU4ahJXm/ | |
| 13:51:31 | mriedem | hmm | |
| 13:51:32 | mriedem | Jun 02 12:46:59.080235 ubuntu-xenial-osic-cloud1-s3500-9104911 nova-conductor[20263]: DEBUG oslo_concurrency.lockutils [None req-a92036cd-3f52-41d6-b104-cb446cfa0ca1 demo demo] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s: NoValidHost_Remote: No valid host was found. There are not enough hosts available. | |
| 13:51:38 | mriedem | that looks like the scheduler is locking on cell0 | |
| 13:52:39 | wy | Sorry sir.If anyone sees my problem,plese help me. I come from China,later,I will go to sleep,It is my sorry. please help me !http://paste.openstack.org/show/X5rwWcSIkukpRU4ahJXm/ | |
| 13:53:51 | mriedem | wy: it's probably better to ask that question on ask.openstack.org, | |
| 13:54:03 | mriedem | yes video playing on the vm is probably causing issues during live migration, | |
| 13:54:11 | mriedem | it would also be helpful to know which version of nova you're running | |
| 13:54:29 | wy | my openstack version is Mikita | |
| 13:55:15 | wy | mriedem nova --version is 3.3.2 | |
| 13:55:20 | wy | is any help? | |
| 13:55:23 | mriedem | wy: fyi https://github.com/openstack/nova/commit/64a482c24d4dfc2aae42672de160ea38e948304c | |
| 13:55:34 | mriedem | ^ is about a known issue in newton, so it's probably also in mitaka | |
| 13:56:03 | wy | thanks..i see | |
| 13:56:17 | mriedem | https://review.openstack.org/#/c/429798/ was the change on master | |