| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-07 | |||
| 18:20:18 | bauzas | https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L558 | |
| 18:20:30 | bauzas | here we error a migration if we have any exception | |
| 18:22:19 | dansmith | right, but after we start the migration, nothing will clean up the destination node's allocations right? | |
| 18:22:33 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9692 | |
| 18:22:37 | bauzas | nah, found it | |
| 18:22:53 | bauzas | we delete the allocation when calling _rollback_live_migration() | |
| 18:23:26 | bauzas | and then we call the drivers's methods for cleaning up the residues... which then unreserve the mdev from the dict :) | |
| 18:23:39 | bauzas | so we're good | |
| 18:24:12 | bauzas | but I agree on adding more logs | |
| 18:24:15 | dansmith | the dict is on the destination, right/ | |
| 18:24:21 | dansmith | so we don't clean that up | |
| 18:24:29 | bauzas | I didn't done that in the past with allocating mdevs and that was needed | |
| 18:24:46 | bauzas | dansmith: no, we both call source and dest virt drivers | |
| 18:24:58 | bauzas | source is here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9708-L9709 | |
| 18:25:03 | dansmith | bauzas: not if rabbit is down | |
| 18:25:10 | bauzas | and dest is there https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9736-L9738 | |
| 18:25:25 | bauzas | dansmith: hah, I get your point now | |
| 18:26:04 | dansmith | just saying it seems like we need more post-disaster recovery bits, be it on compute restart or some periodic | |
| 18:26:18 | bauzas | there could be a situation with rabbit down that would hit the live-migration but we wouldn't unreserve the mdev | |
| 18:26:21 | bauzas | gotcha | |
| 18:26:24 | dansmith | but, given that we do clean these allocations if rabbit is not down and we fail, then fair enough, | |
| 18:26:58 | dansmith | but I think the logging of the internal private data that won't be reset until restart is important, at least so it's discoverable why something is reserved but there's no record of it | |
| 18:27:17 | bauzas | yeah good point | |
| 18:28:06 | bauzas | my kids are starving, so I need to leave or they will run errand in the street like zombies | |
| 18:28:23 | bauzas | but I'll try to be a bit more resilient | |
| 18:28:34 | bauzas | at least the first thing is to log | |
| 18:28:46 | bauzas | then I wonder whether we need some periodic for the cleaning case | |
| 18:29:18 | bauzas | (but I just wonder which conditional to write in that periodic :) ) | |
| 18:29:51 | bauzas | that should be "give me all the migrations that were errored and make sure we don't have a reserved mdev for those" | |
| 18:40:13 | dansmith | something like that | |
| 18:40:43 | sean-k-mooney | melwitt: i think i need to restack but i got https://paste.opendev.org/show/bRUFWOFqOa8kSeXyH12I/ when i tried to delete a vm | |
| 18:42:26 | melwitt | sean-k-mooney: oh, ok that's bc I added a db migration (that I forgot to mention) to add a column (for backing file secret uuid). so you'd have to nova-manage db sync. it's complaining about the lack of the column | |
| 18:43:02 | sean-k-mooney | oh ok | |
| 18:43:06 | sean-k-mooney | i just did a git checkout | |
| 18:43:19 | sean-k-mooney | and restarted things so ya di cna do a db sync | |
| 18:43:35 | melwitt | yeah, I did the same thing. just forgot about the db sync, sorry 😑 | |
| 18:43:58 | sean-k-mooney | i was really confused becasue the way this fails in the openstack client is really terrible | |
| 18:44:29 | sean-k-mooney | venv) ubuntu@disk-encrypt-1:~/repos/devstack$ openstack server delete "test-vm" | |
| 18:44:31 | sean-k-mooney | Resource.get() takes 1 positional argument but 2 were given | |
| 18:44:34 | sean-k-mooney | that is the error you get | |
| 18:44:37 | melwitt | yeah, I remember thinking that too | |
| 18:44:40 | sean-k-mooney | and if you use debug mode it does not help | |
| 18:45:02 | sean-k-mooney | ya so i went back to nova client and it toold me there was a 500 and then it was clear that i was missing something | |
| 18:45:54 | melwitt | guess that's another thing for the osc backlog, figure out what's with that error message and how to improve it | |
| 18:46:47 | sean-k-mooney | this ist its internal traceback https://paste.opendev.org/show/bJQt6QO5X1pdKbL7dnm7/ | |
| 18:46:54 | melwitt | oh, and another thing that tripped me up is the db sync doesn't fan out to cells (!) and you have to use --local_cell to make it sync the nova_cell1 db | |
| 18:47:09 | sean-k-mooney | but the issue is it tyin gto use the resouce before its verified the return code i think | |
| 18:47:48 | sean-k-mooney | good to know cause i totally did not add --local_cell | |
| 18:47:48 | melwitt | hm | |
| 18:48:29 | sean-k-mooney | its still not working for me but it did to the upgrade | |
| 18:48:36 | sean-k-mooney | do i need to als restart | |
| 18:48:55 | sean-k-mooney | Running upgrade 13863f4e1612 -> 2a7173b820a6, Add backing_encryption_secret_uuid to block_device_mapping | |
| 18:48:57 | melwitt | it took me a bit to figure out wtf was wrong too. been that long since I db sync'ed on the fly I guess 😶 | |
| 18:49:41 | melwitt | I don't think you should need to restart | |
| 18:50:10 | melwitt | and make sure you had 'db sync' without --local_cell too bc that's what syncs the nova_cell0 db 😛 | |
| 18:51:18 | melwitt | another thing for the backlog ... do a fanout. I'm not sure if it was deliberate not to or if it's just no one has gotten around to it yet | |
| 18:51:19 | sean-k-mooney | nova-manage --config-dir /etc/nova --config-file /etc/nova/nova_cell1.conf db sync | |
| 18:51:24 | sean-k-mooney | that worked | |
| 18:52:00 | sean-k-mooney | oh you know what this instance was in error so it was burried in cell0 | |
| 18:52:06 | melwitt | hm ok. I thought I tried that but it didn't work without --local_cell | |
| 18:52:47 | melwitt | *didn't work to sync nova_cell1 | |
| 18:53:04 | sean-k-mooney | oh actullly no it got past the schduler so it was in cell 1 | |
| 18:53:20 | sean-k-mooney | i saw was becasue its deleted now so i guess it does not matter | |
| 18:53:44 | sean-k-mooney | also i just booted a vm with your latest code so that cool too | |
| 18:54:09 | sean-k-mooney | ok its on disk-encrypt-2 so in theroy i coudl live migrate it to disk-encrypt-1 | |
| 18:54:48 | melwitt | that should work, but since I said that it probably won't | |
| 18:56:57 | sean-k-mooney | it did not but i havent tried live migration without encypted voluems so i dont know if i missed a step or not | |
| 18:57:13 | sean-k-mooney | ah ok | |
| 18:57:15 | sean-k-mooney | libvirt.libvirtError: Cannot recv data: ssh: Could not resolve hostname disk-encrypt-1: Name or service not known: Connection reset by peer | |
| 18:57:21 | sean-k-mooney | i just need to update /etc/hosts | |
| 18:58:54 | melwitt | 😅 | |
| 19:01:16 | sean-k-mooney | Live Migration failure: Cannot recv data: Host key verification failed.: Connection reset by peer: libvirt.libvirtError: Cannot recv data: Host key verification failed.: Connection reset by peer ok i need to check what usesr its using | |
| 19:01:25 | sean-k-mooney | i see up ubuntu but maybe its runing as root | |
| 19:10:48 | sean-k-mooney | still complaining baout host key verificaiton | |
| 19:10:53 | sean-k-mooney | i think ill just turn that off | |
| 19:22:41 | sean-k-mooney | melwitt: finally i had to swap to using root for some reason for the ssh conenction | |
| 19:22:51 | sean-k-mooney | but in anycase yes live migration in one direction workd | |
| 19:23:15 | sean-k-mooney | im going to check if the instnace dirs ectra actully got cleaned up but looking good so far | |
| 19:24:02 | sean-k-mooney | im so out of partice of doing this with devstack by hand | |
| 19:32:26 | melwitt | sean-k-mooney: yeah, same (I had been) | |
| 19:38:01 | melwitt | sean-k-mooney: oh, with the host key verification in my case what I had to do was both ssh to and from as root "ssh root@blah" but also had to ssh as the other user, maybe the stack user? but pass the root user like "ssh root@blah" | |
| 19:38:27 | melwitt | it would not stop complaining until I did both of those | |
| 20:07:48 | sean-k-mooney | so if i use live_migration_uri = qemu+ssh://root@%s/system | |
| 20:07:58 | sean-k-mooney | and ssh as root to each host | |
| 20:08:01 | sean-k-mooney | then it works fine | |
| 20:08:12 | sean-k-mooney | but i was trying to use live_migration_uri = qemu+ssh://ubuntu@%s/system | |
| 20:08:23 | opendevreview | Ghanshyam proposed openstack/nova master: Remove HyperV: cleanup doc/code ref https://review.opendev.org/c/openstack/nova/+/906629 | |
| 20:08:59 | sean-k-mooney | there is a way to use a non root user to do that but its not worth the hassel | |
| 20:10:13 | opendevreview | Ghanshyam proposed openstack/nova master: HyperV: Remove HyperVLiveMigrateData object https://review.opendev.org/c/openstack/nova/+/906636 | |
| 20:16:28 | melwitt | ah I see | |
| 20:19:46 | opendevreview | Ghanshyam proposed openstack/nova master: HyperV: Remove RDP console connection information API https://review.opendev.org/c/openstack/nova/+/906991 | |
| 20:20:23 | sean-k-mooney | its still unhappy for cold migration so i need to add the root users ssh key to the ubuntu users autherised keys | |
| 20:20:53 | sean-k-mooney | i normally just use one key an put it in all both ubuntu and root with the same public/private key on all hosts | |
| 20:20:56 | opendevreview | Ghanshyam proposed openstack/nova master: HyperV: Remove RDP console connection information API https://review.opendev.org/c/openstack/nova/+/906991 | |
| 20:21:22 | sean-k-mooney | this time i was trying to be fancy and use seperate keys and minimal privladges | |
| 20:21:55 | opendevreview | Ghanshyam proposed openstack/nova master: HyperV: Remove RDP console API https://review.opendev.org/c/openstack/nova/+/906809 | |
| 20:22:16 | melwitt | 🙂 | |
| 20:22:18 | opendevreview | Ghanshyam proposed openstack/nova master: HyperV: Remove extra specs of HyperV driver https://review.opendev.org/c/openstack/nova/+/906992 | |
| 20:28:34 | sean-k-mooney | melwitt: ill try this again tomorrow when i can make sure i set this up correctly end to end | |
| 20:29:09 | sean-k-mooney | of far i have got good coverage of what happens if your ssh keys dont work and we rollback the cold migrate early :) | |