| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-27 | |||
| 12:55:27 | sean-k-mooney | i cant recall what was the cause but i know the env had rabbitmq issuues and network issues | |
| 12:55:28 | sean-k-mooney | i cant recall what was the cause but i know the env had rabbitmq issuues and network issues | |
| 12:58:08 | sean-k-mooney | frickler: the first message by the way is normally (bytes processed=0, remaining=0, total=0) | |
| 12:58:08 | sean-k-mooney | frickler: the first message by the way is normally (bytes processed=0, remaining=0, total=0) | |
| 12:58:33 | sean-k-mooney | then it get updated when the migration actully starts | |
| 12:58:33 | sean-k-mooney | then it get updated when the migration actully starts | |
| 12:59:00 | sean-k-mooney | 2018-04-16 20:46:31.747 767495 INFO nova.virt.libvirt.driver [req-7af65a7d-9d61-4b4b-955d-07d7791b18ac 13515e05a63e48e0b9adb991b250c5f7 8bc196f362b442a4891986517b60389c - - -] [instance: e9c1de25-daa8-43f6-b9b8-a909c19255d9] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) | |
| 12:59:00 | sean-k-mooney | 2018-04-16 20:46:31.747 767495 INFO nova.virt.libvirt.driver [req-7af65a7d-9d61-4b4b-955d-07d7791b18ac 13515e05a63e48e0b9adb991b250c5f7 8bc196f362b442a4891986517b60389c - - -] [instance: e9c1de25-daa8-43f6-b9b8-a909c19255d9] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) | |
| 12:59:02 | sean-k-mooney | 2018-04-16 20:46:37.422 767495 DEBUG nova.virt.libvirt.driver [req-7af65a7d-9d61-4b4b-955d-07d7791b18ac 13515e05a63e48e0b9adb991b250c5f7 8bc196f362b442a4891986517b60389c - - -] [instance: e9c1de25-daa8-43f6-b9b8-a909c19255d9] Migration running for 5 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) _live_migration_monitor | |
| 12:59:02 | sean-k-mooney | 2018-04-16 20:46:37.422 767495 DEBUG nova.virt.libvirt.driver [req-7af65a7d-9d61-4b4b-955d-07d7791b18ac 13515e05a63e48e0b9adb991b250c5f7 8bc196f362b442a4891986517b60389c - - -] [instance: e9c1de25-daa8-43f6-b9b8-a909c19255d9] Migration running for 5 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) _live_migration_monitor | |
| 12:59:04 | sean-k-mooney | /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6428 | |
| 12:59:06 | sean-k-mooney | 2018-04-16 20:46:43.713 767495 DEBUG nova.virt.libvirt.driver [req-7af65a7d-9d61-4b4b-955d-07d7791b18ac 13515e05a63e48e0b9adb991b250c5f7 8bc196f362b442a4891986517b60389c - - -] [instance: e9c1de25-daa8-43f6-b9b8-a909c19255d9] Migration running for 10 secs, memory 0% remaining; (bytes processed=387830565, remaining=0, total=4312604672) _live_migration_monitor | |
| 12:59:06 | sean-k-mooney | 2018-04-16 20:46:43.713 767495 DEBUG nova.virt.libvirt.driver [req-7af65a7d-9d61-4b4b-955d-07d7791b18ac 13515e05a63e48e0b9adb991b250c5f7 8bc196f362b442a4891986517b60389c - - -] [instance: e9c1de25-daa8-43f6-b9b8-a909c19255d9] Migration running for 10 secs, memory 0% remaining; (bytes processed=387830565, remaining=0, total=4312604672) _live_migration_monitor | |
| 12:59:08 | sean-k-mooney | /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6428 | |
| 12:59:08 | sean-k-mooney | /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6428 | |
| 12:59:10 | sean-k-mooney | like htat | |
| 12:59:11 | sean-k-mooney | like htat | |
| 13:00:43 | frickler | in my case, it stays the same for 80000 seconds or so now, migration not finishing at all | |
| 13:00:43 | frickler | in my case, it stays the same for 80000 seconds or so now, migration not finishing at all | |
| 13:01:15 | sean-k-mooney | frickler: do you happen to have ceph? | |
| 13:01:15 | sean-k-mooney | frickler: do you happen to have ceph? | |
| 13:01:38 | frickler | sean-k-mooney: for volume storage, yes. root disk is local | |
| 13:01:38 | frickler | sean-k-mooney: for volume storage, yes. root disk is local | |
| 13:01:54 | sean-k-mooney | that was form https://bugzilla.redhat.com/show_bug.cgi?id=1566723 | |
| 13:01:55 | openstack | bugzilla.redhat.com bug 1566723 in RBD "Instance live migration times out after <x> seconds with nova - doesn't copy instance memory" [High,Closed: errata] - Assigned to jdillama | |
| 13:01:55 | sean-k-mooney | that was form https://bugzilla.redhat.com/show_bug.cgi?id=1566723 | |
| 13:01:55 | openstack | bugzilla.redhat.com bug 1566723 in RBD "Instance live migration times out after <x> seconds with nova - doesn't copy instance memory" [High,Closed: errata] - Assigned to jdillama | |
| 13:03:08 | sean-k-mooney | frickler: apparently restarting OSDs durign the migration could cause locking issues in librbd in the past | |
| 13:03:08 | sean-k-mooney | frickler: apparently restarting OSDs durign the migration could cause locking issues in librbd in the past | |
| 13:03:33 | frickler | oh, I can check whether this is actually a boot-from-volume instance | |
| 13:03:33 | frickler | oh, I can check whether this is actually a boot-from-volume instance | |
| 13:04:20 | sean-k-mooney | frickler: there is also https://bugs.launchpad.net/nova/+bug/1583145 | |
| 13:04:21 | openstack | Launchpad bug 1644248 in OpenStack Compute (nova) "duplicate for #1583145 Nova incorrectly tracks live migration progress" [High,Fix released] | |
| 13:04:21 | sean-k-mooney | frickler: there is also https://bugs.launchpad.net/nova/+bug/1583145 | |
| 13:04:21 | openstack | Launchpad bug 1644248 in OpenStack Compute (nova) "duplicate for #1583145 Nova incorrectly tracks live migration progress" [High,Fix released] | |
| 13:04:44 | sean-k-mooney | oh the other one is fixed | |
| 13:04:44 | sean-k-mooney | oh the other one is fixed | |
| 13:04:49 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1644248 | |
| 13:04:50 | openstack | Launchpad bug 1644248 in OpenStack Compute (nova) "Nova incorrectly tracks live migration progress" [High,Fix released] | |
| 13:04:50 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1644248 | |
| 13:04:50 | openstack | Launchpad bug 1644248 in OpenStack Compute (nova) "Nova incorrectly tracks live migration progress" [High,Fix released] | |
| 13:18:13 | sean-k-mooney | bauzas: gibi lyarwood if we were to impleemnte the generic nova manage command for updating image metadata | |
| 13:18:13 | sean-k-mooney | bauzas: gibi lyarwood if we were to impleemnte the generic nova manage command for updating image metadata | |
| 13:18:26 | sean-k-mooney | how would ye feel about backpoarting hat | |
| 13:18:26 | sean-k-mooney | how would ye feel about backpoarting hat | |
| 13:18:28 | sean-k-mooney | *that | |
| 13:18:28 | sean-k-mooney | *that | |
| 13:18:57 | sean-k-mooney | elod: any tought form a stable core perspective | |
| 13:18:57 | sean-k-mooney | elod: any tought form a stable core perspective | |
| 13:19:44 | sean-k-mooney | context is we have a custoemr downstream that really want to fix a few instance by adding os-type=windows to the inatnce that were booted before they fixed there windows images | |
| 13:19:45 | sean-k-mooney | context is we have a custoemr downstream that really want to fix a few instance by adding os-type=windows to the inatnce that were booted before they fixed there windows images | |
| 13:20:18 | sean-k-mooney | and today they will have to do a manual db update to do that | |
| 13:20:18 | sean-k-mooney | and today they will have to do a manual db update to do that | |
| 13:20:38 | sean-k-mooney | then live/cold migrate teh inststances so they land up in the correct host aggrate after | |
| 13:20:38 | sean-k-mooney | then live/cold migrate teh inststances so they land up in the correct host aggrate after | |
| 13:21:38 | bauzas | sean-k-mooney: in general, we don't accept to backport features like this | |
| 13:21:39 | bauzas | sean-k-mooney: in general, we don't accept to backport features like this | |
| 13:21:42 | bauzas | (upstream) | |
| 13:21:42 | bauzas | (upstream) | |
| 13:22:23 | sean-k-mooney | bauzas: in general but we have backported nova-manage command on the bases it resolved operator pain in the past | |
| 13:22:24 | sean-k-mooney | bauzas: in general but we have backported nova-manage command on the bases it resolved operator pain in the past | |
| 13:22:35 | sean-k-mooney | like the heal allocations ones | |
| 13:22:35 | sean-k-mooney | like the heal allocations ones | |
| 13:28:33 | bauzas | sean-k-mooney: yup, because it's a bugfix | |
| 13:28:33 | bauzas | sean-k-mooney: yup, because it's a bugfix | |
| 13:29:49 | bauzas | but fwiw, we abandoned the backports for the placement audit https://review.opendev.org/q/I537ed74503d208957f0a97af3ab754a6750dac20 | |
| 13:29:49 | bauzas | but fwiw, we abandoned the backports for the placement audit https://review.opendev.org/q/I537ed74503d208957f0a97af3ab754a6750dac20 | |
| 13:41:45 | elod | sean-k-mooney: bauzas is right: in general, we don't accept to backport features like this | |
| 13:41:46 | elod | sean-k-mooney: bauzas is right: in general, we don't accept to backport features like this | |
| 13:43:18 | frickler | sean-k-mooney: turns out this migration was in status "queued", being held up by another one in progress that is having trouble getting the disk synced, seems I didn't look close enough there. thanks for your help anyway | |
| 13:43:18 | frickler | sean-k-mooney: turns out this migration was in status "queued", being held up by another one in progress that is having trouble getting the disk synced, seems I didn't look close enough there. thanks for your help anyway | |
| 13:43:59 | sean-k-mooney | frickler: glad you figured it out | |
| 13:43:59 | sean-k-mooney | frickler: glad you figured it out | |
| 13:45:16 | sean-k-mooney | elod: bauzas yep i know and in general we do not allow custoemr to do db udate eitehr. i dont partically want to do a downstream only backport of that feature. i think it would be generic useful but if we do not want to backport it that is fine | |
| 13:45:16 | sean-k-mooney | elod: bauzas yep i know and in general we do not allow custoemr to do db udate eitehr. i dont partically want to do a downstream only backport of that feature. i think it would be generic useful but if we do not want to backport it that is fine | |
| 14:09:50 | elod | sean-k-mooney: well, according to policy, there could be exceptions. I still tend to say no for this backport (though I haven't seen the implementation) | |
| 14:09:50 | elod | sean-k-mooney: well, according to policy, there could be exceptions. I still tend to say no for this backport (though I haven't seen the implementation) | |
| 14:10:22 | elod | exception* -- if stable cores agree... | |
| 14:10:22 | elod | exception* -- if stable cores agree... | |
| 14:10:51 | sean-k-mooney | elod: its not written yet :) | |
| 14:10:51 | sean-k-mooney | elod: its not written yet :) | |
| 14:12:49 | sean-k-mooney | elod: it would be https://review.opendev.org/c/openstack/nova/+/774896/8/nova/cmd/manage.py excpet instead of setting just machine_type you could set any image metatadata value that does not chagne resouce usage | |
| 14:12:49 | sean-k-mooney | elod: it would be https://review.opendev.org/c/openstack/nova/+/774896/8/nova/cmd/manage.py excpet instead of setting just machine_type you could set any image metatadata value that does not chagne resouce usage | |
| 14:13:57 | sean-k-mooney | elod: anyway its fine i was going to say if it was short to implement and contovertial then maybe instead of deliving a custom sql srcript we could implement this and backport it for our customer to use | |
| 14:13:57 | sean-k-mooney | elod: anyway its fine i was going to say if it was short to implement and contovertial then maybe instead of deliving a custom sql srcript we could implement this and backport it for our customer to use | |
| 14:19:00 | gibi | elod, sean-k-mooney: we discussed backport nova-manage "feature" in the past. Like heal allocation. It is useful, it is self contained in mostly in the manage.py and it is not impacting any of our public APIs | |
| 14:19:00 | gibi | elod, sean-k-mooney: we discussed backport nova-manage "feature" in the past. Like heal allocation. It is useful, it is self contained in mostly in the manage.py and it is not impacting any of our public APIs | |
| 14:19:22 | gibi | so it is pretty safe to backport | |
| 14:19:22 | gibi | so it is pretty safe to backport | |
| 14:19:48 | gibi | I was lazy and did not backported the port allocation part of the heal allocation command to stable but we did agreed with mriedem in the past that that could happen | |
| 14:19:48 | gibi | I was lazy and did not backported the port allocation part of the heal allocation command to stable but we did agreed with mriedem in the past that that could happen | |
| 14:20:39 | gibi | also I'm supporting to backporting thigs that helps avoiding the need of direct and manual db update | |
| 14:20:39 | gibi | also I'm supporting to backporting thigs that helps avoiding the need of direct and manual db update | |
| 14:21:12 | gibi | nobody likes customer bugs where the db was manually changed for some reason and it caused inconsitencies | |
| 14:21:12 | gibi | nobody likes customer bugs where the db was manually changed for some reason and it caused inconsitencies | |
| 14:25:48 | elod | in that case... if it really is as gibi says, and the implementation looks OK in the sense of backportability... then so be it :] | |
| 14:25:48 | elod | in that case... if it really is as gibi says, and the implementation looks OK in the sense of backportability... then so be it :] | |
| 14:26:14 | elod | if there are no objections from other stable cores | |
| 14:26:15 | elod | if there are no objections from other stable cores | |
| 14:26:55 | gibi | :) | |