| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-05 | |||
| 13:51:31 | mriedem | hmm | |
| 13:51:32 | mriedem | Jun 02 12:46:59.080235 ubuntu-xenial-osic-cloud1-s3500-9104911 nova-conductor[20263]: DEBUG oslo_concurrency.lockutils [None req-a92036cd-3f52-41d6-b104-cb446cfa0ca1 demo demo] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s: NoValidHost_Remote: No valid host was found. There are not enough hosts available. | |
| 13:51:38 | mriedem | that looks like the scheduler is locking on cell0 | |
| 13:52:39 | wy | Sorry sir.If anyone sees my problem,plese help me. I come from China,later,I will go to sleep,It is my sorry. please help me !http://paste.openstack.org/show/X5rwWcSIkukpRU4ahJXm/ | |
| 13:53:51 | mriedem | wy: it's probably better to ask that question on ask.openstack.org, | |
| 13:54:03 | mriedem | yes video playing on the vm is probably causing issues during live migration, | |
| 13:54:11 | mriedem | it would also be helpful to know which version of nova you're running | |
| 13:54:29 | wy | my openstack version is Mikita | |
| 13:55:15 | wy | mriedem nova --version is 3.3.2 | |
| 13:55:20 | wy | is any help? | |
| 13:55:23 | mriedem | wy: fyi https://github.com/openstack/nova/commit/64a482c24d4dfc2aae42672de160ea38e948304c | |
| 13:55:34 | mriedem | ^ is about a known issue in newton, so it's probably also in mitaka | |
| 13:56:03 | wy | thanks..i see | |
| 13:56:17 | mriedem | https://review.openstack.org/#/c/429798/ was the change on master | |
| 13:56:21 | mriedem | try disabling that config option | |
| 13:56:33 | wy | but is any ways to solve? | |
| 13:56:49 | wy | ok..i will have a try | |
| 13:56:57 | mriedem | please see the change and related bug report | |
| 13:57:27 | mriedem | mordred: i've got to get my kid to school and then will continue looking at this - i can't see any other changes around friday that would have caused this except the one yo'uve already identified | |
| 13:57:33 | wy | ok...you are so kind | |
| 13:57:47 | mriedem | there is likely some tightly coupled gorp in the scheduler that i'm not aware of | |
| 13:58:03 | mordred | mriedem: awesome - and thanks - I'll keep poking too | |
| 14:00:58 | edleafe | Scheduler subteam meeting starting now in #openstack-meeting-alt | |
| 14:04:04 | wy | mriedem hi,I have seen the url you send.However,My migration is live-migration and is based on the NFS.Importantly,I can not find the configuration--timeout...what should I do? | |
| 14:06:13 | wy | mriedem sorry..I have found.I will try,if have any results,I will tell you | |
| 14:16:55 | openstackgerrit | Sean Dague proposed openstack/nova master: Send request_id on glance calls https://review.openstack.org/467242 | |
| 14:21:55 | wy | I have a question. live-migration speed has any relation to the band and others? | |
| 14:22:25 | artom | wy, you mean bandwidth? Yes | |
| 14:22:39 | artom | It's essentially copying the VM's RAM from one host to the other | |
| 14:23:06 | wy | artom just bandwidth? | |
| 14:23:13 | prometheanfire | ocata python-novaclient still caps pbr, can we get a release to start uncapping again? | |
| 14:24:09 | artom | wy, not only. The rate at which the contents of RAM changes also affects it | |
| 14:24:34 | mriedem | prometheanfire: sure, will do | |
| 14:24:35 | wy | Now the problem has none.But,I just open a video on a VM but not play it,now it has migrated 5minutes but not over | |
| 14:24:42 | artom | The faster the source memory is changing, the longer it'll take, since the destination has to "catch up" by receiving more memory | |
| 14:25:41 | wy | but the VM is just opening a video not playing...But why it takes so long? | |
| 14:26:03 | mriedem | prometheanfire: need to get https://review.openstack.org/#/c/464867/ in first | |
| 14:26:16 | wy | mriedem thanks for you.I have solved problem as you said..but the time it takes so long | |
| 14:26:35 | artom | wy, sometimes the best things in life take time ;) | |
| 14:26:45 | wy | artom ok.. | |
| 14:27:00 | wy | artom I do not understand | |
| 14:27:10 | artom | wy, just a joke, ignore it :) | |
| 14:27:32 | wy | artom the vm ram is not changing now..but it takes so long | |
| 14:27:50 | artom | wy, I'm not sure what else you need. You said mriedem's suggestion solved your problem. | |
| 14:27:50 | prometheanfire | mriedem: k, thanks | |
| 14:28:27 | mriedem | artom: always the romantic | |
| 14:28:32 | wy | artom yes...I changed the configuration.just disable the live_migration_progress_timeout | |
| 14:29:07 | mriedem | wy: the question is, does it eventually complete | |
| 14:29:13 | mriedem | now that you've disabled that timeout | |
| 14:29:58 | wy | mriedem now,it is still migrating for about 8minutes ,it does not have error | |
| 14:30:07 | artom | mriedem, if you have a moment, mind taking a look at https://review.openstack.org/#/c/465205/? | |
| 14:30:11 | wy | mriedem could I think it effects? | |
| 14:30:39 | mriedem | artom: would like lyarwood_ and mdbooth to look at that first, | |
| 14:30:43 | artom | Fairly straightforward fix to the XML that swap_volume generates | |
| 14:30:48 | mriedem | they've been involved in swap volume stuff in the libvirt driver for awhile | |
| 14:30:50 | artom | mriedem, they did, on a previous revision | |
| 14:31:05 | mriedem | i see | |
| 14:31:06 | artom | Nothing's really changed | |
| 14:31:15 | artom | Comments, and formatting, and... | |
| 14:31:16 | mriedem | well i've got to dig into mordred's problem first | |
| 14:31:28 | artom | mriedem, no, my stuff is obviously way more important | |
| 14:31:29 | mriedem | i can star it | |
| 14:31:32 | wy | mriedem but i think it would not migrate successfully forever | |
| 14:31:35 | mordred | artom: :) | |
| 14:32:03 | mriedem | wy: i may be mistaken but i though there was a way to check progress with virsh or qemu command line | |
| 14:32:13 | mordred | mriedem: I just pushed up a silly patch in shade to look for "Successfully synced instances from host" in the scheduler logs before starting to run shade's tests | |
| 14:32:50 | wy | mriedem i have found none.openstack has not provide command to see the process | |
| 14:33:52 | wy | mriedem i am seesing the compute log.but the remaining data decends for a while then asend.. | |
| 14:34:14 | wy | mriedem could I think it would not migrate successfully forever? | |
| 14:34:56 | mriedem | wy: i mean like "migrate -d URI" https://www.linux-kvm.org/page/Migration#Management | |
| 14:36:16 | mriedem | wy: https://www.youtube.com/watch?v=Vy8rS_5vea4 about 9 minutes in talks about some of this | |
| 14:36:23 | mriedem | virsh domjobinfo <domain> | |
| 14:36:25 | mriedem | for example | |
| 14:36:36 | wy | mriedem thanks,i will have a look at | |
| 14:36:44 | wy | mriedem you are so helper | |
| 14:37:46 | mordred | mriedem: because looking througha few logs of successful runs, that seems to happen approximately around when we launch functional tests anyway - so it may just be a legit race-condition between the cloud settling in and the tests getting started (although also maybe there is a real bug obvs) | |
| 14:37:48 | wy | mriedem sorry | |
| 14:37:55 | wy | mriedem i do not understand | |
| 14:38:03 | wy | mriedem but what is the uri? | |
| 14:38:06 | mordred | mriedem: also, every time someone types "virsh" I think they're mistyping vishy ... | |
| 14:38:14 | wy | mriedem what i should input in the command | |
| 14:39:56 | mriedem | wy: virsh domjobinfo --help ? | |
| 14:40:03 | mriedem | i assume <domain> is the domain id | |
| 14:40:10 | wy | mriedem It is still migrating about many minutes..I just shut down the video.it successfully migrate | |
| 14:40:12 | mriedem | e.g. 00000000ef or something | |
| 14:40:17 | wy | mriedem ok | |
| 14:40:27 | wy | mriedem is the video affects? | |
| 14:41:20 | mriedem | wy: i assume so yes | |
| 14:41:33 | mriedem | everything the source vm is doing has to be transferred to the destination vm | |
| 14:42:10 | wy | mriedem but i just open the video but not playing it ..it means the video is suspend | |
| 14:43:33 | wy | mriedem open the video but not playing it,this will contribute to the change of the ram? | |
| 14:44:13 | wy | mriedem or we can say,my bandwidth is not enough,so the changes of ram is bigger than the transmit?? | |
| 14:47:30 | wy | mriedem thanks very much.it is 11,i have to go to dormitory to sleep.It is my pleasure to make friends with you.You are so kind | |
| 14:48:01 | wy | mriedem but ,If you have seen,please help me solve my questions...ahah | |
| 14:50:06 | openstackgerrit | Andrey Volkov proposed openstack/nova master: [WIP] [placement] Add api-ref for allocations https://review.openstack.org/470933 | |
| 14:50:06 | openstackgerrit | Andrey Volkov proposed openstack/nova master: [placement] Add api-ref for aggregates https://review.openstack.org/470932 | |
| 14:50:12 | mriedem | wy: you're welcome. i don't have a good answer to your question about a paused video during a live migration | |
| 14:50:23 | mriedem | and this isn't really the place to debug that level of detail probably | |
| 14:51:06 | mriedem | mordred: ok so back to your thing, this stacktrace is when the compute starts up and calls update_instance_info before the host is mapped http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/screen-n-sch.txt.gz#_Jun_02_12_45_15_755877 | |
| 14:51:14 | mriedem | then we map the host here http://logs.openstack.org/95/468095/2/gate/gate-shade-functional/3f5d8dc/logs/devstacklog.txt.gz#_2017-06-02_12_46_19_540 | |