| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-10 | |||
| 19:08:06 | melwitt | (as opposed to counting unmigrated and being potentially wrong in the count) | |
| 19:09:52 | mriedem | right i gathered that from the exchange between you and surya but i hadn't looked at the code yet that does the fallback logic | |
| 19:12:14 | melwitt | oh, I see | |
| 19:12:51 | melwitt | I misunderstood what you meant by saying you needed to load the context of how it will be used | |
| 19:20:04 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Last Level Cache https://review.openstack.org/651233 | |
| 19:37:31 | efried_rollin | mriedem: done | |
| 19:51:09 | melwitt | efried, mriedem: working on this old service uuid upgrade bug, IIUC because of this change https://review.openstack.org/620711 if we do a service delete followed by a service create, the running compute service will never create the resource provider and compute node again, for the same 'host', because it knows about it already in the RT | |
| 19:51:39 | melwitt | even after another update_available_resource interval | |
| 19:52:07 | efried | hm, I thought someone fixed that recently. | |
| 19:52:19 | melwitt | the func tests (written before that change landed) now fail because can't get the RP created again | |
| 19:53:16 | efried | oh, this is if you delete and recreate the service without actually stopping the running process? | |
| 19:53:17 | melwitt | I rebased the set today, so I should have all of the latest changes | |
| 19:53:22 | melwitt | correct | |
| 19:53:40 | efried | Yes, this was definitely addressed recently, but I thought someone killed the change for reasons. Let me find it... | |
| 19:53:54 | mriedem | https://review.openstack.org/#/c/641899/ ? | |
| 19:54:46 | efried | Yup, that's the one. melwitt ^ | |
| 19:54:50 | melwitt | looking | |
| 19:55:42 | melwitt | I'm not even getting to that code though, it looks like that would at least try to create the provider | |
| 19:55:53 | efried | melwitt: Does that have the wrong bug number associated with it? | |
| 19:56:08 | efried | (I'm asking, why didn't you find it if you were looking into this already?) | |
| 19:56:32 | efried | duplicate bugs? | |
| 19:56:46 | melwitt | oh they're saying after a service restart the RP create fails | |
| 19:57:03 | melwitt | efried: oh sorry, I'm working on refreshing patches for an unrelated bug | |
| 19:57:30 | melwitt | and ran into this problem of the RP will never be created again while the service is running | |
| 19:57:50 | melwitt | because the func tests I'm working with are now failing because of it | |
| 19:58:27 | efried | I mean, it makes sense, because we have the provider tree cache, and no reason to think it has expired, so we never go looking for the provider to notice it's gone so we never create it. | |
| 19:58:37 | efried | This is what SIGHUP was supposed to be for. | |
| 19:59:06 | melwitt | ok, so it's a known thing that if an operator deletes and creates a service with the same hostname, they're supposed to SIGHUP now | |
| 19:59:07 | efried | And this would be a doc issue: "If you delete and recreate the service, SIGHUP (or restart ffs) your n-cpu process" | |
| 19:59:52 | efried | Well, I'm saying if it's not documented as such, that would be my first proposed solution. I'm not positive it's going to fix the problem (even when SIGHUP works, which it still doesn't afaik) but it'd be the first thing I'd try. | |
| 19:59:58 | mriedem | fwiw we already have a note about that in the API reference https://developer.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service | |
| 20:00:02 | melwitt | ok. I can simulate that in the test, but wanted to mention it in case it wasn't desired/expected | |
| 20:00:20 | melwitt | thanks | |
| 20:00:53 | efried | cool, the doc actually says to stop the thing, which makes freakin sense to me. | |
| 20:01:11 | efried | Maybe I'm being too simplistic in my thinking | |
| 20:01:23 | efried | but how does an operator expect "delete a thing but keep it running" to ever work? | |
| 20:01:36 | melwitt | no, it's fine, I just saw a change and didn't have the context about it | |
| 20:02:14 | melwitt | well, the delete it and create it again, so they expect it to be running after they create it, in this scenario | |
| 20:02:14 | efried | sorry, tbc I'm not griping at you; I guess I'm kind of wtf-ing that there's a bug about this. | |
| 20:02:29 | efried | Perhaps we should have delete kill the service | |
| 20:02:39 | efried | I think that may be what cdent was suggesting in the above patch | |
| 20:03:05 | melwitt | I can't remember why operators have done that, something to do with an upgrade. cfriesen ran into it before I think | |
| 20:03:36 | efried | ...or make the delete operation fail if the service is running, to force the operator to do it. | |
| 20:03:42 | melwitt | (delete service, create service, for same hostname) | |
| 20:04:03 | efried | Oh, I'm *sure* there's a good reason to do that ^ | |
| 20:04:05 | efried | :) | |
| 20:04:34 | efried | but expecting the running process to survive intact and work properly through it... | |
| 20:05:09 | efried | perhaps it used to work by pure luck and now they're facing RBB. | |
| 20:06:07 | cfriesen | I think we probably hit it during upgrade...our upgrade mechanism was to delete the standby controller, then reinstall it with the new load (and the same hostname as before), then migrate the DB over and cut over to the new controller. | |
| 20:07:11 | melwitt | cfriesen: that sounds like what I recall | |
| 20:07:48 | cfriesen | hmm..that wouldn't delete the nova-compute service though. | |
| 20:08:36 | melwitt | cfriesen: yeah this was deleting the nova-compute service for a host, then recreating it. deleting the nova-compute service record would kill the compute node record too | |
| 20:08:53 | melwitt | and you're saying your upgrade routine would not do that | |
| 20:10:43 | cfriesen | I don't think so...trying to find the context for https://bugs.launchpad.net/nova/+bug/1764556 | |
| 20:10:45 | openstack | Launchpad bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 20:11:25 | melwitt | cfriesen: it's ok, don't spend time on it. we were just talking about the whole service delete and create thing. behavior changed semi recently | |
| 20:11:42 | melwitt | and I thought I remembered you might have done something like that in the past | |
| 20:11:52 | cfriesen | okay, so it looks like for that bug we *did* delete a compute node while on Newton, then migrated to Pike, then created a new node with the same name. May have been an artificial test though, not actually required for upgrade. | |
| 20:12:06 | cfriesen | upgraded to Pike, rather | |
| 20:12:12 | melwitt | gotcha | |
| 20:16:13 | mriedem | the delete compute service while it's still running was never really thought out i don't think | |
| 20:16:23 | mriedem | we've only semi recently tried to shore up a lot of that nonsense | |
| 20:16:47 | mriedem | it's only been noticed as more of a problem since we started building onto it with things like host mappings and resource providers | |
| 20:17:33 | mriedem | i.e. this is semi recent https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L247 | |
| 20:17:41 | mriedem | and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L270 | |
| 20:17:46 | mriedem | and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L275 | |
| 20:18:30 | melwitt | ah yeah, I remember that first one | |
| 20:23:54 | openstackgerrit | Merged openstack/nova stable/stein: doc: Capitalize keystone domain name https://review.openstack.org/650600 | |
| 20:24:00 | openstackgerrit | Merged openstack/nova stable/stein: Add functional regression test for bug 1669054 https://review.openstack.org/649319 | |
| 20:24:01 | openstack | bug 1669054 in OpenStack Compute (nova) stein "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem) | |
| 20:24:08 | openstackgerrit | Merged openstack/nova stable/stein: doc: Fix openstack CLI command https://review.openstack.org/648412 | |
| 20:28:06 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556 https://review.openstack.org/562041 | |
| 20:28:07 | openstack | bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt) | |
| 20:28:07 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional regression test for bug 1778305 https://review.openstack.org/582407 | |
| 20:28:08 | openstackgerrit | melanie witt proposed openstack/nova master: Don't generate service UUID for deleted services https://review.openstack.org/582408 | |
| 20:28:09 | openstack | bug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt) | |
| 20:31:46 | openstackgerrit | Chris Friesen proposed openstack/nova stable/rocky: Add missing libvirt exception during device detach https://review.openstack.org/651637 | |
| 20:34:12 | openstackgerrit | Chris Friesen proposed openstack/nova stable/queens: Add missing libvirt exception during device detach https://review.openstack.org/651639 | |
| 20:36:46 | openstackgerrit | Chris Friesen proposed openstack/nova stable/pike: Add missing libvirt exception during device detach https://review.openstack.org/651642 | |
| 20:54:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use InstanceList.get_count_by_hosts when deleting a compute service https://review.openstack.org/651647 | |
| 21:12:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650 | |
| 21:15:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650 | |
| 21:33:08 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: FUP for I68498afd481f7291a6102928d7999b4be49ded7a https://review.openstack.org/651653 | |
| 21:47:09 | openstackgerrit | Merged openstack/nova stable/stein: Override the 'get' method in DriverBlockDevice class https://review.openstack.org/647646 | |
| 21:47:14 | openstackgerrit | Merged openstack/nova stable/stein: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/650060 | |
| 21:55:55 | openstackgerrit | Merged openstack/nova stable/stein: libvirt: disconnect volume when encryption fails https://review.openstack.org/650931 | |
| 22:01:33 | openstackgerrit | Merged openstack/nova stable/stein: Move create of ComputeAPI object in websocketproxy https://review.openstack.org/649374 | |
| 22:35:13 | openstackgerrit | Colleen Murphy proposed openstack/nova stable/rocky: Move create of ComputeAPI object in websocketproxy https://review.openstack.org/649375 | |
| 23:51:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Revert "Wait for network-vif-plugged on resize revert" https://review.openstack.org/639396 | |
| 23:51:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Revert resize: wait for external events in compute manager https://review.openstack.org/644881 | |
| #openstack-nova - 2019-04-11 | |||
| 00:34:03 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Base Enablement https://review.openstack.org/651130 | |
| 01:39:41 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Expose auto converge and post copy https://review.openstack.org/651681 | |
| 02:42:17 | gmann | sean-k-mooney: artom : we are good on 648123 from microversion point of view. It is not breaking any API contract. Commented inline. https://review.openstack.org/#/c/648123/4 | |
| 02:42:43 | gmann | tempest schema test is passing which make sure the API contract verification | |
| 03:11:37 | openstackgerrit | Boxiang Zhu proposed openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | |
| 03:54:37 | openstackgerrit | Merged openstack/nova master: Add test coverage for nova.privsep.libvirt. https://review.openstack.org/648616 | |
| 04:06:22 | openstackgerrit | Merged openstack/nova master: Add test coverage for nova.privsep.qemu. https://review.openstack.org/649191 | |
| 04:16:07 | openstackgerrit | Merged openstack/nova stable/stein: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/649320 | |
| 09:02:52 | openstackgerrit | Theodoros Tsioutsias proposed openstack/nova-specs master: Add PENDING vm state https://review.openstack.org/648687 | |
| 10:20:00 | openstackgerrit | Chris Dent proposed openstack/nova master: Use update_provider_tree in vmware virt driver https://review.openstack.org/651615 | |