| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-12 | |||
| 14:05:04 | dswebb | Hi,have a bit of an odd issue with Livemigration on rocky. I've got a 3 node nova cluster using iscsi -> nimble as my storage. I can successfully migrate a VM from node 1 to node 2 (or node2 to node1) once. If I try and migrate the vm back to the node it started on (or any node it's been on prior) it fails silently to migrate and the only logs I see that suggest any issue are the following: https://pastebin.com/Y2ccf3MR | |
| 14:05:54 | gibi | mriedem: sorry I have to get back to you tomorrow (my) morning | |
| 14:06:36 | mriedem | gibi: ok i'll just post the change since it'll be simple and i need to rebase the series anyway, if you have a problem with it you and dan can fight to the death on the review | |
| 14:07:18 | gibi | mriedem: sure. works for me :) | |
| 14:07:58 | kashyap | Ah, efried is gone ... just when I have his answer for him | |
| 14:15:39 | luyao | efried: hi Eric, in case of you not seeing the msg just now, I need you confirm if I need a blueprint and separate spec for vpmem live migration. Thanks in advance. | |
| 14:16:57 | dswebb | also worth mentioning that the same live migration (back and forth) using ceph as the cinder block store works fine | |
| 14:18:14 | efried | luyao: I don't know; let's put it on the agenda for the next meeting and ask the team. | |
| 14:19:04 | luyao | efried: OK, thanks | |
| 14:19:06 | efried | luyao: I think gibi had at least a blueprint and possibly a spec for the continuation of lifecycle ops with bandwidth providers, but I think that's a much bigger effort than for vpmem. | |
| 14:19:34 | kashyap | efried: Hey, so on your encrypted emulated TPM question -- | |
| 14:20:22 | kashyap | efried: The QEMU version does not matter for encryption. What matters is the 'swtpm'. And you need at least libvirt 5.6.0 for the 'encryption' element | |
| 14:20:25 | kashyap | https://libvirt.org/formatdomain.html#elementsTpm | |
| 14:21:29 | efried | kashyap: sweet, thank you very much. I was aware of the libvirt side, but couldn't find the qemu side documented anywhere. | |
| 14:21:58 | efried | bauzas, gibi: Thank you both for writing up those summaries! | |
| 14:22:08 | bauzas | np | |
| 14:22:10 | bauzas | HTH | |
| 14:23:18 | luyao | efried: yes, in terms of code size, the lm for vpmem is not big | |
| 14:24:43 | efried | luyao: was there mention of the details of lm implementation in the original spec? (Save me digging it up) | |
| 14:25:54 | luyao | efried: no, and I need update the original spec, since our implementation is different from it | |
| 14:27:36 | openstackgerrit | George proposed openstack/nova stable/ocata: Support qemu >= 2.10 https://review.opendev.org/693851 | |
| 14:28:13 | efried | luyao: I think a short spec is probably a good idea, since (IIRC) there are RPC object changes, right? | |
| 14:29:42 | luyao | efried: yes, there is | |
| 14:30:12 | efried | So yeah, please put together a brief spec. Thanks. | |
| 14:31:47 | luyao | efried: Get it, thanks | |
| 15:04:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 15:04:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Helper to start computes with different HostInfos https://review.opendev.org/686832 | |
| 15:04:26 | openstack | bug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms) | |
| 15:04:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146 https://review.opendev.org/687404 | |
| 15:05:31 | artom | stephenfin, if you've recovered from PTG and still have all your organs, ^^ is still waiting :) | |
| 15:07:17 | bauzas | artom: stephenfin is still somewhere in China | |
| 15:07:33 | artom | bauzas, intentionally? ;) | |
| 15:08:01 | dansmith | lyarwood: I reviewed your rescue spec.. do you have any idea what needs to happen and or what blockers there would be to making rescue work for BFV instances? | |
| 15:08:08 | bauzas | artom: and I can't tell for his organs, but last time I saw him, his liver was working correctly | |
| 15:08:09 | bauzas | :p | |
| 15:08:38 | artom | bauzas, haha | |
| 15:08:47 | bauzas | artom: well, he found that Shanghai wasn't that huge so he visited another city | |
| 15:09:06 | bauzas | oh, and FWIW, I'll keep my wechat account | |
| 15:09:30 | bauzas | living in China for one week makes you realize how life can be different without big G and a few other tools | |
| 15:14:05 | lyarwood | dansmith: thanks for that, yeah I have a rough idea but I've not looked at this since failing to land this in Newton. I can't think of much that's changed since then with regards to the high level implementation. I also can't think of any additional blockers for BFV at the moment tbh. | |
| 15:15:05 | dansmith | lyarwood: okay, maybe I'll try removing the BFV check on a devstack and poke around with what is and isn't working in the current setup | |
| 15:15:22 | dansmith | lyarwood: because BFV instances are some of the more important ones you'd want to rescue, ya'know | |
| 15:16:11 | lyarwood | dansmith: right, so for rescue to work you need to switch things around so the rescue device is the final thing attached, not the first and then boot from that device. | |
| 15:17:03 | dansmith | lyarwood: that's what you're proposing.... | |
| 15:19:35 | lyarwood | dansmith: right sorry I think I'm misunderstanding what you're actually asking here. | |
| 15:20:16 | dansmith | lyarwood: right now we just categorically refuse to rescue a BFV instance, and I think we should change that since BFV instances are likely pets, and likely in need of rescue | |
| 15:20:30 | dansmith | so, with your proposed changes, in addition to the ordering, also including cinder volumes, | |
| 15:20:47 | dansmith | I'm wondering what we could do to follow up your changes with a change to allow rescue of BFV instances | |
| 15:21:34 | smcginnis | I've seen cattle with BFV too. Some just prefer to keep their storage on Cinder volumes. FWIW. | |
| 15:22:00 | dansmith | smcginnis: sure I know | |
| 15:22:22 | dansmith | smcginnis: I said "likely" not "always" :) | |
| 15:23:36 | lyarwood | dansmith: Ah! The changes would actually allow BFV and non-BFV but I can see that isn't made clear in the spec. I totally missed this when copying it over from an older change this morning but I'll clean this up and make it clear now. | |
| 15:24:02 | dansmith | lyarwood: sweet | |
| 15:24:13 | dansmith | lyarwood: do you have old crusty code of up for this at all? | |
| 15:24:58 | lyarwood | dansmith: there's an old abandoned series but iirc that was missing the compute service check that I think I had somewhere locally | |
| 15:25:11 | dansmith | lyarwood: ack | |
| 15:25:43 | dansmith | lyarwood: so you think that after these changes, if the api let them, we could rescue BFV instances no problem? | |
| 15:25:43 | lyarwood | dansmith: https://review.opendev.org/#/q/status:abandoned+topic:bp/virt-rescue-stable-disk-devices | |
| 15:26:05 | lyarwood | dansmith: yes | |
| 15:26:12 | dansmith | if so, we'll need a compute capability to indicate this support, and could use that in place of the version check | |
| 15:26:57 | lyarwood | yup true, I also didn't cover microversions in the spec that I think came up in that series | |
| 15:29:24 | dansmith | lyarwood: no REST API changes, so why a microversion? | |
| 15:31:56 | lyarwood | dansmith: https://review.opendev.org/#/c/270288/18/nova/compute/api.py@2777 - mriedem raised it here as something we might have to do, I'm also not sure if that's still the case tbh. | |
| 15:32:22 | dansmith | lyarwood: oh, you even remove that in the code, I see, | |
| 15:32:46 | dansmith | well, in that case it probably does need a microversion and a rest api impact section, but in the spec I just reviewed you said "none" :) | |
| 15:34:02 | lyarwood | yeah apologies this slipped my mind when I was pushing it up for review earlier. I'll mark the spec as WIP for now while I get things back in order. | |
| 15:34:52 | dansmith | lyarwood: you said you were going to work on this this cycle right? | |
| 15:35:52 | lyarwood | dansmith: yup I should actually have time this cycle | |
| 15:36:10 | dansmith | lyarwood: cool, I | |
| 15:36:23 | dansmith | I would suggest getting this going sooner than later if you can schedule it | |
| 15:36:39 | dansmith | at least rebase the code and get it restored so that others can comment/help | |
| 15:37:01 | dansmith | even if before the spec, although I think the spec is easy to merge pretty soon if you fix up the issues | |
| 15:37:33 | lyarwood | dansmith: sure, I should be able to get something rebased pretty quickly. | |
| 15:37:36 | lyarwood | famous last words | |
| 15:41:03 | efried | jroll: randomly generated passphrase: do you think there should be a conf option for the length? If so/not, what should the default/number be? | |
| 15:49:24 | jroll | efried: probably not worth a config until/unless someone asks. 32 or 64 characters seems reasonable | |
| 15:50:19 | efried | Roger that. (Glad I asked, I was gonna go 256 :P ) | |
| 15:50:37 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Wire up a force disconnect_volume flag https://review.opendev.org/584849 | |
| 15:50:37 | jroll | 256 is also reasonable :D | |
| 15:51:18 | jroll | efried: I guess if nobody ever touches this, maybe 256 is a better option | |
| 15:51:47 | efried | it certainly won't be readable/typable. Like an ssh key. How big are those? | |
| 15:52:45 | jroll | $ wc ~/.ssh/id_rsa | |
| 15:52:47 | jroll | 54 61 3326 /Users/jrollenhagen/.ssh/id_rsa | |
| 15:53:43 | efried | yours is bigger than mine | |
| 15:53:53 | dansmith | mriedem: wha-chu-think main? https://review.opendev.org/#/c/636224/50/nova/compute/api.py | |
| 15:53:59 | jroll | heh | |
| 15:54:01 | dansmith | efried: whoa, wrong channel | |
| 15:54:44 | efried | that's number of chars, b64-encoded, plus header/footer | |
| 15:55:42 | jroll | right, looks like 3072 bytes | |
| 15:55:51 | mriedem | dansmith: you can't compare None values can you? | |
| 15:56:41 | dansmith | mriedem: you can | |
| 15:57:13 | dansmith | True | |
| 15:57:13 | dansmith | mriedem: >>> (None, 1) < (None, 2) | |
| 15:57:26 | mriedem | TypeError: can't compare datetime.datetime to NoneType | |
| 15:57:26 | mriedem | File "<stdin>", line 1, in <module> | |
| 15:57:26 | mriedem | Traceback (most recent call last): | |
| 15:57:26 | mriedem | >>> datetime.datetime.today() < None | |
| 15:57:37 | mriedem | i guess because of the tuples | |
| 15:57:42 | bauzas | AFAIR, you were even able to say something like None = 2 | |
| 15:57:43 | dansmith | oh you mean compare datetime to none | |