Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-10
17:19:30 dansmith mnaser: yeah, that could be a thing, but isn't currently
17:19:49 mnaser fair enough, so for now probably best to stick to parsing CLI output
17:20:25 dansmith mnaser: tbh, I think you probably don't want to import and run in your own namespace, since talking to cells requires pivoting global oslo.db state and other things
17:21:38 mnaser dansmith: fair enough.. so probably better off just doing CLI operations
17:21:56 dansmith mnaser: the cli commands aim to be pretty idempotent, but if we need --robot output formatting or something, then we should do that
17:26:33 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use update_provider_tree in vmware virt driver https://review.openstack.org/651615
17:45:36 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Nova LLC allocation - RMD plugin for RDT CAT https://review.openstack.org/651233
17:56:12 mriedem dansmith: it's a perf improvement to avoid a call to get allocations for the instance created by the scheduler when we already have those in the Selection object in scope
17:56:29 dansmith mriedem: ..right I got that ;)
17:56:48 dansmith is that really substantial enough to backport?
17:58:44 mriedem it's just new code in stein and missed the cutoff so it's not controversial imo
17:58:49 mriedem we wouldn't die without it no
17:59:21 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Nova local resource management that uses RMD https://review.openstack.org/651130
18:57:32 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Base Enablement https://review.openstack.org/651130
19:06:32 mriedem melwitt: i re-read the counting quotas from placement spec for train again and noticed a few things that could be cleaned up https://review.openstack.org/#/c/645302/
19:06:59 melwitt mriedem: ok, thanks
19:07:40 melwitt mriedem: to your other comment (I'll reply on the review too), we're not counting unmigrated qfd instance mappings because we're falling back to legacy counting if unmigrated qfd instance mappings are detected
19:08:06 melwitt (as opposed to counting unmigrated and being potentially wrong in the count)
19:09:52 mriedem right i gathered that from the exchange between you and surya but i hadn't looked at the code yet that does the fallback logic
19:12:14 melwitt oh, I see
19:12:51 melwitt I misunderstood what you meant by saying you needed to load the context of how it will be used
19:20:04 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Last Level Cache https://review.openstack.org/651233
19:37:31 efried_rollin mriedem: done
19:51:09 melwitt efried, mriedem: working on this old service uuid upgrade bug, IIUC because of this change https://review.openstack.org/620711 if we do a service delete followed by a service create, the running compute service will never create the resource provider and compute node again, for the same 'host', because it knows about it already in the RT
19:51:39 melwitt even after another update_available_resource interval
19:52:07 efried hm, I thought someone fixed that recently.
19:52:19 melwitt the func tests (written before that change landed) now fail because can't get the RP created again
19:53:16 efried oh, this is if you delete and recreate the service without actually stopping the running process?
19:53:17 melwitt I rebased the set today, so I should have all of the latest changes
19:53:22 melwitt correct
19:53:40 efried Yes, this was definitely addressed recently, but I thought someone killed the change for reasons. Let me find it...
19:53:54 mriedem https://review.openstack.org/#/c/641899/ ?
19:54:46 efried Yup, that's the one. melwitt ^
19:54:50 melwitt looking
19:55:42 melwitt I'm not even getting to that code though, it looks like that would at least try to create the provider
19:55:53 efried melwitt: Does that have the wrong bug number associated with it?
19:56:08 efried (I'm asking, why didn't you find it if you were looking into this already?)
19:56:32 efried duplicate bugs?
19:56:46 melwitt oh they're saying after a service restart the RP create fails
19:57:03 melwitt efried: oh sorry, I'm working on refreshing patches for an unrelated bug
19:57:30 melwitt and ran into this problem of the RP will never be created again while the service is running
19:57:50 melwitt because the func tests I'm working with are now failing because of it
19:58:27 efried I mean, it makes sense, because we have the provider tree cache, and no reason to think it has expired, so we never go looking for the provider to notice it's gone so we never create it.
19:58:37 efried This is what SIGHUP was supposed to be for.
19:59:06 melwitt ok, so it's a known thing that if an operator deletes and creates a service with the same hostname, they're supposed to SIGHUP now
19:59:07 efried And this would be a doc issue: "If you delete and recreate the service, SIGHUP (or restart ffs) your n-cpu process"
19:59:52 efried Well, I'm saying if it's not documented as such, that would be my first proposed solution. I'm not positive it's going to fix the problem (even when SIGHUP works, which it still doesn't afaik) but it'd be the first thing I'd try.
19:59:58 mriedem fwiw we already have a note about that in the API reference https://developer.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service
20:00:02 melwitt ok. I can simulate that in the test, but wanted to mention it in case it wasn't desired/expected
20:00:20 melwitt thanks
20:00:53 efried cool, the doc actually says to stop the thing, which makes freakin sense to me.
20:01:11 efried Maybe I'm being too simplistic in my thinking
20:01:23 efried but how does an operator expect "delete a thing but keep it running" to ever work?
20:01:36 melwitt no, it's fine, I just saw a change and didn't have the context about it
20:02:14 efried sorry, tbc I'm not griping at you; I guess I'm kind of wtf-ing that there's a bug about this.
20:02:14 melwitt well, the delete it and create it again, so they expect it to be running after they create it, in this scenario
20:02:29 efried Perhaps we should have delete kill the service
20:02:39 efried I think that may be what cdent was suggesting in the above patch
20:03:05 melwitt I can't remember why operators have done that, something to do with an upgrade. cfriesen ran into it before I think
20:03:36 efried ...or make the delete operation fail if the service is running, to force the operator to do it.
20:03:42 melwitt (delete service, create service, for same hostname)
20:04:03 efried Oh, I'm *sure* there's a good reason to do that ^
20:04:05 efried :)
20:04:34 efried but expecting the running process to survive intact and work properly through it...
20:05:09 efried perhaps it used to work by pure luck and now they're facing RBB.
20:06:07 cfriesen I think we probably hit it during upgrade...our upgrade mechanism was to delete the standby controller, then reinstall it with the new load (and the same hostname as before), then migrate the DB over and cut over to the new controller.
20:07:11 melwitt cfriesen: that sounds like what I recall
20:07:48 cfriesen hmm..that wouldn't delete the nova-compute service though.
20:08:36 melwitt cfriesen: yeah this was deleting the nova-compute service for a host, then recreating it. deleting the nova-compute service record would kill the compute node record too
20:08:53 melwitt and you're saying your upgrade routine would not do that
20:10:43 cfriesen I don't think so...trying to find the context for https://bugs.launchpad.net/nova/+bug/1764556
20:10:45 openstack Launchpad bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] - Assigned to melanie witt (melwitt)
20:11:25 melwitt cfriesen: it's ok, don't spend time on it. we were just talking about the whole service delete and create thing. behavior changed semi recently
20:11:42 melwitt and I thought I remembered you might have done something like that in the past
20:11:52 cfriesen okay, so it looks like for that bug we *did* delete a compute node while on Newton, then migrated to Pike, then created a new node with the same name. May have been an artificial test though, not actually required for upgrade.
20:12:06 cfriesen upgraded to Pike, rather
20:12:12 melwitt gotcha
20:16:13 mriedem the delete compute service while it's still running was never really thought out i don't think
20:16:23 mriedem we've only semi recently tried to shore up a lot of that nonsense
20:16:47 mriedem it's only been noticed as more of a problem since we started building onto it with things like host mappings and resource providers
20:17:33 mriedem i.e. this is semi recent https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L247
20:17:41 mriedem and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L270
20:17:46 mriedem and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L275
20:18:30 melwitt ah yeah, I remember that first one
20:23:54 openstackgerrit Merged openstack/nova stable/stein: doc: Capitalize keystone domain name https://review.openstack.org/650600
20:24:00 openstackgerrit Merged openstack/nova stable/stein: Add functional regression test for bug 1669054 https://review.openstack.org/649319
20:24:01 openstack bug 1669054 in OpenStack Compute (nova) stein "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem)
20:24:08 openstackgerrit Merged openstack/nova stable/stein: doc: Fix openstack CLI command https://review.openstack.org/648412
20:28:06 openstackgerrit melanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556 https://review.openstack.org/562041
20:28:07 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1778305 https://review.openstack.org/582407
20:28:07 openstack bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)
20:28:08 openstackgerrit melanie witt proposed openstack/nova master: Don't generate service UUID for deleted services https://review.openstack.org/582408
20:28:09 openstack bug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)
20:31:46 openstackgerrit Chris Friesen proposed openstack/nova stable/rocky: Add missing libvirt exception during device detach https://review.openstack.org/651637
20:34:12 openstackgerrit Chris Friesen proposed openstack/nova stable/queens: Add missing libvirt exception during device detach https://review.openstack.org/651639
20:36:46 openstackgerrit Chris Friesen proposed openstack/nova stable/pike: Add missing libvirt exception during device detach https://review.openstack.org/651642
20:54:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Use InstanceList.get_count_by_hosts when deleting a compute service https://review.openstack.org/651647
21:12:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650
21:15:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650
21:33:08 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: FUP for I68498afd481f7291a6102928d7999b4be49ded7a https://review.openstack.org/651653

Earlier   Later