| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-05 | |||
| 13:41:12 | wy | morning every one! | |
| 13:41:13 | wy | ~ | |
| 13:41:14 | mriedem | i thought we used to call discover_hosts or simple_cell_setup in here https://github.com/openstack-dev/devstack/blob/master/lib/nova | |
| 13:41:18 | mriedem | but i'm not seeing that now | |
| 13:41:25 | wy | I do the live-migration successfully.However,when the VM(centos7) is playing video,the migration is failed.But,if I shut down the video,the migration is successful. | |
| 13:41:50 | wy | does the video affect the migration? | |
| 13:42:37 | wy | 2017-06-05 20:34:23.970 32069 ERROR nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Live Migration failure: operation aborted: migration job: canceled by client | |
| 13:42:37 | wy | 2017-06-05 20:34:23.664 32069 WARNING nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Live migration stuck for 150 sec | |
| 13:42:37 | wy | INFO nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Data remaining 63950848 bytes, low watermark 729088 bytes 143 seconds ago | |
| 13:42:40 | wy | 2017-06-05 20:34:24.170 32069 ERROR nova.virt.libvirt.driver [req-b6f2476e-c9bc-42bd-9c34-9db9e5db0543 bb012ac6b0534fdba5bf70a16785fd8b 3bf434c0309549a291a79b38b34ed7e5 - - -] [instance: dc891233-dac6-4b9a-a1eb-fb51de84fddd] Migration operation has aborted | |
| 13:42:43 | wy | 2017-06-05 20:34:40.872 32069 INFO nova.compute.resource_tracker [req-36ecb5cc-4060-4006-8ed4-5c196020a583 - - - - -] Auditing locally available compute resources for node compute1 | |
| 13:43:19 | mordred | wy: hi! when you have things like that to share, please use http://paste.openstack.org/ and past a link instead | |
| 13:44:11 | mordred | mriedem: discover_hosts seems to be top level now | |
| 13:45:14 | edleafe | Scheduler subteam meeting in 15 minutes in #openstack-meeting-alt | |
| 13:45:39 | mordred | mriedem: http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/devstacklog.txt.gz#_2017-06-02_12_46_16_568 | |
| 13:46:33 | mriedem | 2017-06-02 12:46:19.540 | Creating host mapping for compute host 'ubuntu-xenial-osic-cloud1-s3500-9104911': bc5499e9-2e12-4ae6-9e5b-58b8104de163 | |
| 13:46:44 | mriedem | mordred: that's what we're waiting on - does your failed test run before that? | |
| 13:47:00 | mriedem | looks like the test runs after | |
| 13:47:19 | mriedem | Jun 02 12:46:58.103925 ubuntu-xenial-osic-cloud1-s3500-9104911 nova-api[16069]: INFO nova.osapi_compute.wsgi.server [None req-a92036cd-3f52-41d6-b104-cb446cfa0ca1 demo demo] 10.39.244.211 "POST /v2.1/servers HTTP/1.1" status: 202 len: 870 time: 0.6169720 | |
| 13:47:44 | wy | ok..thankd | |
| 13:49:17 | mriedem | and we blow up here http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-cond.txt.gz#_Jun_02_12_46_59_071485 | |
| 13:49:24 | wy | http://paste.openstack.org/ | |
| 13:49:53 | efried | mriedem mordred Haven't looked at the bug itself, but it sounds like it could be related to something we hit in the PowerVM CI, where discover_hosts was running before the compute service was ready to handle it. esberglu added a --strict option to discover_hosts that makes it exit nonzero unless it actually registered your host, so you can run it in a loop until it succeeds. | |
| 13:50:42 | wy | http://paste.openstack.org/show/X5rwWcSIkukpRU4ahJXm/ | |
| 13:51:31 | mriedem | hmm | |
| 13:51:32 | mriedem | Jun 02 12:46:59.080235 ubuntu-xenial-osic-cloud1-s3500-9104911 nova-conductor[20263]: DEBUG oslo_concurrency.lockutils [None req-a92036cd-3f52-41d6-b104-cb446cfa0ca1 demo demo] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s: NoValidHost_Remote: No valid host was found. There are not enough hosts available. | |
| 13:51:38 | mriedem | that looks like the scheduler is locking on cell0 | |
| 13:52:39 | wy | Sorry sir.If anyone sees my problem,plese help me. I come from China,later,I will go to sleep,It is my sorry. please help me !http://paste.openstack.org/show/X5rwWcSIkukpRU4ahJXm/ | |
| 13:53:51 | mriedem | wy: it's probably better to ask that question on ask.openstack.org, | |
| 13:54:03 | mriedem | yes video playing on the vm is probably causing issues during live migration, | |
| 13:54:11 | mriedem | it would also be helpful to know which version of nova you're running | |
| 13:54:29 | wy | my openstack version is Mikita | |
| 13:55:15 | wy | mriedem nova --version is 3.3.2 | |
| 13:55:20 | wy | is any help? | |
| 13:55:23 | mriedem | wy: fyi https://github.com/openstack/nova/commit/64a482c24d4dfc2aae42672de160ea38e948304c | |
| 13:55:34 | mriedem | ^ is about a known issue in newton, so it's probably also in mitaka | |
| 13:56:03 | wy | thanks..i see | |
| 13:56:17 | mriedem | https://review.openstack.org/#/c/429798/ was the change on master | |
| 13:56:21 | mriedem | try disabling that config option | |
| 13:56:33 | wy | but is any ways to solve? | |
| 13:56:49 | wy | ok..i will have a try | |
| 13:56:57 | mriedem | please see the change and related bug report | |
| 13:57:27 | mriedem | mordred: i've got to get my kid to school and then will continue looking at this - i can't see any other changes around friday that would have caused this except the one yo'uve already identified | |
| 13:57:33 | wy | ok...you are so kind | |
| 13:57:47 | mriedem | there is likely some tightly coupled gorp in the scheduler that i'm not aware of | |
| 13:58:03 | mordred | mriedem: awesome - and thanks - I'll keep poking too | |
| 14:00:58 | edleafe | Scheduler subteam meeting starting now in #openstack-meeting-alt | |
| 14:04:04 | wy | mriedem hi,I have seen the url you send.However,My migration is live-migration and is based on the NFS.Importantly,I can not find the configuration--timeout...what should I do? | |
| 14:06:13 | wy | mriedem sorry..I have found.I will try,if have any results,I will tell you | |
| 14:16:55 | openstackgerrit | Sean Dague proposed openstack/nova master: Send request_id on glance calls https://review.openstack.org/467242 | |
| 14:21:55 | wy | I have a question. live-migration speed has any relation to the band and others? | |
| 14:22:25 | artom | wy, you mean bandwidth? Yes | |
| 14:22:39 | artom | It's essentially copying the VM's RAM from one host to the other | |
| 14:23:06 | wy | artom just bandwidth? | |
| 14:23:13 | prometheanfire | ocata python-novaclient still caps pbr, can we get a release to start uncapping again? | |
| 14:24:09 | artom | wy, not only. The rate at which the contents of RAM changes also affects it | |
| 14:24:34 | mriedem | prometheanfire: sure, will do | |
| 14:24:35 | wy | Now the problem has none.But,I just open a video on a VM but not play it,now it has migrated 5minutes but not over | |
| 14:24:42 | artom | The faster the source memory is changing, the longer it'll take, since the destination has to "catch up" by receiving more memory | |
| 14:25:41 | wy | but the VM is just opening a video not playing...But why it takes so long? | |
| 14:26:03 | mriedem | prometheanfire: need to get https://review.openstack.org/#/c/464867/ in first | |
| 14:26:16 | wy | mriedem thanks for you.I have solved problem as you said..but the time it takes so long | |
| 14:26:35 | artom | wy, sometimes the best things in life take time ;) | |
| 14:26:45 | wy | artom ok.. | |
| 14:27:00 | wy | artom I do not understand | |
| 14:27:10 | artom | wy, just a joke, ignore it :) | |
| 14:27:32 | wy | artom the vm ram is not changing now..but it takes so long | |
| 14:27:50 | prometheanfire | mriedem: k, thanks | |
| 14:27:50 | artom | wy, I'm not sure what else you need. You said mriedem's suggestion solved your problem. | |
| 14:28:27 | mriedem | artom: always the romantic | |
| 14:28:32 | wy | artom yes...I changed the configuration.just disable the live_migration_progress_timeout | |
| 14:29:07 | mriedem | wy: the question is, does it eventually complete | |
| 14:29:13 | mriedem | now that you've disabled that timeout | |
| 14:29:58 | wy | mriedem now,it is still migrating for about 8minutes ,it does not have error | |
| 14:30:07 | artom | mriedem, if you have a moment, mind taking a look at https://review.openstack.org/#/c/465205/? | |
| 14:30:11 | wy | mriedem could I think it effects? | |
| 14:30:39 | mriedem | artom: would like lyarwood_ and mdbooth to look at that first, | |
| 14:30:43 | artom | Fairly straightforward fix to the XML that swap_volume generates | |
| 14:30:48 | mriedem | they've been involved in swap volume stuff in the libvirt driver for awhile | |
| 14:30:50 | artom | mriedem, they did, on a previous revision | |
| 14:31:05 | mriedem | i see | |
| 14:31:06 | artom | Nothing's really changed | |
| 14:31:15 | artom | Comments, and formatting, and... | |
| 14:31:16 | mriedem | well i've got to dig into mordred's problem first | |
| 14:31:28 | artom | mriedem, no, my stuff is obviously way more important | |
| 14:31:29 | mriedem | i can star it | |
| 14:31:32 | wy | mriedem but i think it would not migrate successfully forever | |
| 14:31:35 | mordred | artom: :) | |
| 14:32:03 | mriedem | wy: i may be mistaken but i though there was a way to check progress with virsh or qemu command line | |
| 14:32:13 | mordred | mriedem: I just pushed up a silly patch in shade to look for "Successfully synced instances from host" in the scheduler logs before starting to run shade's tests | |
| 14:32:50 | wy | mriedem i have found none.openstack has not provide command to see the process | |
| 14:33:52 | wy | mriedem i am seesing the compute log.but the remaining data decends for a while then asend.. | |
| 14:34:14 | wy | mriedem could I think it would not migrate successfully forever? | |
| 14:34:56 | mriedem | wy: i mean like "migrate -d URI" https://www.linux-kvm.org/page/Migration#Management | |
| 14:36:16 | mriedem | wy: https://www.youtube.com/watch?v=Vy8rS_5vea4 about 9 minutes in talks about some of this | |
| 14:36:23 | mriedem | virsh domjobinfo <domain> | |
| 14:36:25 | mriedem | for example | |
| 14:36:36 | wy | mriedem thanks,i will have a look at | |
| 14:36:44 | wy | mriedem you are so helper | |
| 14:37:46 | mordred | mriedem: because looking througha few logs of successful runs, that seems to happen approximately around when we launch functional tests anyway - so it may just be a legit race-condition between the cloud settling in and the tests getting started (although also maybe there is a real bug obvs) | |