| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 12:08:37 | amorin | hey all | |
| 12:08:52 | amorin | I am facing issue on my openstack deployment when I try to live migrate an instance | |
| 12:09:00 | amorin | if the original image is deleted on glance | |
| 12:09:05 | amorin | live-migration fail | |
| 12:09:09 | amorin | I am using openstack newton | |
| 12:09:23 | amorin | do you know if this bug is already declared somewhere ? | |
| 12:09:26 | amorin | Eventually fixed? | |
| 13:09:34 | bauzas | amorin: do you have some stacktrace to share ? | |
| 13:10:17 | amorin | well, it's not a stacktrace but just a image not found | |
| 13:10:20 | amorin | I will show you | |
| 13:10:54 | amorin | I have that on source host: | |
| 13:10:57 | amorin | [instance: e4c16231-5c3e-49d3-b985-c963bfa52437] Live Migration failure: internal error: info migration reply was missing return status | |
| 13:11:00 | bauzas | amorin: we had a bug like this https://bugs.launchpad.net/nova/+bug/1270825 | |
| 13:11:00 | openstack | Launchpad bug 1270825 in OpenStack Compute (nova) "Live block migration fails for instances whose glance images have been deleted" [High,Fix released] - Assigned to melanie witt (melwitt) | |
| 13:11:59 | amorin | sounds like my issue ! | |
| 13:12:13 | bauzas | we also had https://bugs.launchpad.net/nova/+bug/1546778 | |
| 13:12:13 | openstack | Launchpad bug 1546778 in OpenStack Compute (nova) liberty "libvirt: resize with deleted backing image fails" [Medium,Fix released] - Assigned to Matthew Booth (mbooth-9) | |
| 13:12:36 | bauzas | but that's for resize | |
| 13:12:56 | amorin | first bug look good, but I am running openstack newton... | |
| 13:13:04 | amorin | seems that the bug is supposed to be fixed since juno | |
| 13:13:37 | bauzas | yup, very old bug, hence the needed stacktrace | |
| 13:13:49 | bauzas | we need to understand more why it fails | |
| 13:17:26 | amorin | I'll try to restart nova compute in debug and find a trace | |
| 13:17:31 | amorin | on both source and dest host | |
| 13:17:41 | amorin | I will come back to you asap | |
| 13:20:32 | bauzas | ok, please highlight me when you reply so I can see it | |
| 13:20:48 | bauzas | amorin: like this :) | |
| 13:20:59 | amorin | yup | |
| 13:22:36 | openstackgerrit | Adam Spiers proposed openstack/nova-specs master: Add spec for libvirt driver launching AMD SEV-encrypted instances https://review.openstack.org/609779 | |
| 13:42:08 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Handle missing marker during online data migration https://review.openstack.org/610974 | |
| 13:45:18 | mnaser | asking here because i think nova would probably be a project that does this but | |
| 13:45:25 | mnaser | is it possible to have multiple rootwrap daemons? | |
| 13:45:53 | mnaser | i'm running to problems with neutron vpnagent doing a lot of commands that take a long time which block everything else from running (in rootwrap daemon) | |
| 13:46:18 | dansmith | multiple in a load-balance situation? I kinda doubt it | |
| 13:46:32 | mnaser | even as in dedicated situation | |
| 13:46:33 | dansmith | I think that's one of the many performance limitations of rootwrap | |
| 13:46:40 | mnaser | it also made me wonder if privsep has the same issues | |
| 13:46:45 | mnaser | which i think it might not? | |
| 13:46:57 | dansmith | I think privsep is the same | |
| 13:47:02 | mnaser | because i frequently see privsep spawn processes for a specific module or so | |
| 13:47:08 | dansmith | ISTR the cinder people being concerned about that | |
| 13:47:32 | mnaser | i much rather have vpnaas (in this case) fight for resources between itself rather than with things that are important and need to happen in a few seconds | |
| 13:49:27 | mnaser | i guess the advantage this would give would be the ability to replace a shell out by a code/library | |
| 13:52:56 | smcginnis | There is a performance issue with privsep right now that it serializes anything it runs. | |
| 13:53:11 | smcginnis | So only one "priveleged" thing can happen at a time. | |
| 13:53:19 | smcginnis | But there's a patch up to fix that. | |
| 13:53:22 | mnaser | smcginnis: thats the case with rootwrap daemon too, no? | |
| 13:53:32 | mnaser | at least, that's what the behavior im seeing anyways | |
| 13:53:49 | smcginnis | mnaser: I didn't think so. That just calls out to run commands, so I thought it didn't have the same issue. | |
| 13:54:14 | mnaser | well, rootwrap yes, it just calls out to run commads, but rootwrap daemon seems to do the whole serialize thing | |
| 13:54:24 | mnaser | (i think) | |
| 13:54:33 | smcginnis | Here's the privsep patch if anyone is interested - https://review.openstack.org/#/c/593556/ | |
| 13:55:01 | smcginnis | It must not be quite as bad. There was push back on moving fully to privsep because there was a noticeable performance impact in doing so. | |
| 13:55:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 13:57:36 | dansmith | smcginnis: to be clear, he's talking about rootwrap *daemon* | |
| 14:00:21 | smcginnis | Yeah | |
| 14:01:38 | mnaser | i wonder if i can spawn an independent/second rootwrap daemon | |
| 14:19:26 | bauzas | mnaser: I guess the problem is how nova.rootwrap would know which daemon to pick | |
| 14:21:41 | bauzas | mnaser: oh wait, you can spawn multiple daemons, each per service, nope ? | |
| 14:21:58 | mnaser | bauzas: i think so, thats what im attempting | |
| 14:22:01 | mnaser | just launch another client.. | |
| 14:22:04 | openstackgerrit | Merged openstack/nova master: Use tempest-pg-full https://review.openstack.org/609954 | |
| 14:22:47 | bauzas | anyway, taxi time | |
| 14:57:33 | bauzas | mnaser: interesting to read https://specs.openstack.org/openstack/oslo-specs/specs/juno/rootwrap-daemon-mode.html#client-api | |
| 15:05:39 | bauzas | mnaser: more interesting https://github.com/openstack/nova/blob/master/nova/utils.py#L126 | |
| 15:05:53 | bauzas | mnaser: we allow one client per rootwrap config | |
| 15:06:07 | bauzas | if two configs, two clients | |
| 15:06:19 | bauzas | and since clients lazily load daemons if needed... | |
| 15:07:29 | bauzas | and I guess https://github.com/openstack/nova/blob/master/nova/utils.py#L123 is your PITA | |
| 15:09:39 | mnaser | bauzas: I’m investigating on how to pull this out and seeing what breaks terribly with multiple clients | |
| 15:13:53 | bauzas | mnaser: could you test something ? what if you have two distinct services running different config files, with each of them differencing by the rootwrap_config option value | |
| 15:14:17 | bauzas | mnaser: in this case, I guess we would automatically create two clients and two daemons | |
| 15:27:24 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955 | |
| 15:29:47 | kashyap | mriedem: Hi, I saw a ping fly by last night on 'hpet' and libvirt. I wonder if it's resolved | |
| 15:34:42 | mriedem | https://review.openstack.org/#/c/607989/ | |
| 15:35:05 | mriedem | lots of chatter yesterday about whether or not libvirt exposed hpet capability as a clock source for the host caps | |
| 15:35:09 | mriedem | it doesn't look like it does | |
| 15:36:50 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Do not call linux_net.delete_net_dev on Windows https://review.openstack.org/610916 | |
| 15:36:51 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Fix random test_unplug_ovs failures https://review.openstack.org/611017 | |
| 15:36:59 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: VMware: OVA and StrOpt images as VM templates https://review.openstack.org/609736 | |
| 15:50:25 | mriedem | gerritbot must be dead | |
| 15:50:32 | mriedem | imacdonn: +W on https://review.openstack.org/#/c/608091/ | |
| 15:51:13 | imacdonn | mriedem: OK, thanks .... I need to get a new rev in to fix a typo on in the release note | |
| 15:51:34 | mriedem | i fixed it | |
| 15:51:50 | imacdonn | mriedem: ok, cool, thanks .. just got email from review too | |
| 16:00:31 | kashyap | mriedem: So, from a quick chat w/ the QEMU & libvirt folks -- | |
| 16:01:06 | kashyap | They say: "I'd would not do that" (configuring 'hpet') | |
| 16:01:12 | kashyap | As it's super expensive compared to other timer sources | |
| 16:01:18 | kashyap | And even worse for virtual machines | |
| 16:01:50 | kashyap | "HPET access involves a context switch to QEMU userspace, where as TSC is handled by KVM natively" | |
| 16:02:06 | kashyap | mriedem: But to your original question -- no I don't see it in libvirt's host capabilities either. | |
| 16:04:40 | kashyap | Probably because "no one has asked for it before" | |
| 16:12:29 | kashyap | mriedem: Responded on the review with details. | |
| 16:14:35 | efried | hah, so after all of that, we may wind up not doing this thing at all? | |
| 16:15:35 | cdent | rad | |
| 16:15:59 | mriedem | well it was explicitly disabled in libvirt guests originally for a reason | |
| 16:16:11 | mriedem | b/c apparently at least for windows images it can skew the clock in the guest | |
| 16:16:25 | mriedem | kashyap: thanks for investigating | |
| 16:16:34 | mriedem | artom: +2 on your live migration cleanup thing https://review.openstack.org/#/c/609517/ | |
| 16:16:48 | artom | mriedem, thanks for the thorough reviewing :) | |