| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-10 | |||
| 17:09:37 | dansmith | mriedem: am I missing what bug this actually fixes? https://review.openstack.org/#/c/647713/1 | |
| 17:09:44 | sean-k-mooney | the intended worflow being edit on host then docker restart nova_compute | |
| 17:10:01 | cfriesen | sean-k-mooney: yeah, that'd be cleaner | |
| 17:12:50 | cdent | cfriesen: is this sort of live code tweaking a thing you find common? | |
| 17:13:10 | cdent | I mean I can imagine it often enough in test settings, experiments, and the like | |
| 17:13:37 | cdent | but in a deployed world aren't the choices usually: observe, kill, make a new one? | |
| 17:14:09 | mnaser | there's not really any pythonic api to manage cells, right? | |
| 17:14:14 | mnaser | or any api at all | |
| 17:15:37 | mnaser | I am looking at adding multicell support in openstackansible, one of the things to introduce is the addition of some way to define cells | |
| 17:15:50 | cfriesen | cdent: this would be for tweaking stuff on lab systems. I agree for areas where devstack is sufficient that'd be better. | |
| 17:18:14 | cdent | cfriesen: I guess I was thinking a lab system is one of the main areas where "you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image..." wouldn't be too much of a big deal, and you might have some simple(-ish) tooling to support it | |
| 17:18:27 | cdent | I agree it is a PITA | |
| 17:18:43 | dansmith | mnaser: you mean via the external API I assume? there isn't by design | |
| 17:19:08 | dansmith | mnaser: or do you mean something like being able to import nova-manage and use its routines? | |
| 17:19:15 | mnaser | dansmith: yeah.. or I was thinking maybe we could do something like.. import nova.cmd.manage; manage.add_cell() or whatever | |
| 17:19:21 | cfriesen | cdent: tool improvements would help, I agree. | |
| 17:19:26 | mnaser | #2 (because ansible can just have python scripts run) | |
| 17:19:30 | dansmith | mnaser: yeah, that could be a thing, but isn't currently | |
| 17:19:49 | mnaser | fair enough, so for now probably best to stick to parsing CLI output | |
| 17:20:25 | dansmith | mnaser: tbh, I think you probably don't want to import and run in your own namespace, since talking to cells requires pivoting global oslo.db state and other things | |
| 17:21:38 | mnaser | dansmith: fair enough.. so probably better off just doing CLI operations | |
| 17:21:56 | dansmith | mnaser: the cli commands aim to be pretty idempotent, but if we need --robot output formatting or something, then we should do that | |
| 17:26:33 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use update_provider_tree in vmware virt driver https://review.openstack.org/651615 | |
| 17:45:36 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Nova LLC allocation - RMD plugin for RDT CAT https://review.openstack.org/651233 | |
| 17:56:12 | mriedem | dansmith: it's a perf improvement to avoid a call to get allocations for the instance created by the scheduler when we already have those in the Selection object in scope | |
| 17:56:29 | dansmith | mriedem: ..right I got that ;) | |
| 17:56:48 | dansmith | is that really substantial enough to backport? | |
| 17:58:44 | mriedem | it's just new code in stein and missed the cutoff so it's not controversial imo | |
| 17:58:49 | mriedem | we wouldn't die without it no | |
| 17:59:21 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Nova local resource management that uses RMD https://review.openstack.org/651130 | |
| 18:57:32 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Base Enablement https://review.openstack.org/651130 | |
| 19:06:32 | mriedem | melwitt: i re-read the counting quotas from placement spec for train again and noticed a few things that could be cleaned up https://review.openstack.org/#/c/645302/ | |
| 19:06:59 | melwitt | mriedem: ok, thanks | |
| 19:07:40 | melwitt | mriedem: to your other comment (I'll reply on the review too), we're not counting unmigrated qfd instance mappings because we're falling back to legacy counting if unmigrated qfd instance mappings are detected | |
| 19:08:06 | melwitt | (as opposed to counting unmigrated and being potentially wrong in the count) | |
| 19:09:52 | mriedem | right i gathered that from the exchange between you and surya but i hadn't looked at the code yet that does the fallback logic | |
| 19:12:14 | melwitt | oh, I see | |
| 19:12:51 | melwitt | I misunderstood what you meant by saying you needed to load the context of how it will be used | |
| 19:20:04 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Last Level Cache https://review.openstack.org/651233 | |
| 19:37:31 | efried_rollin | mriedem: done | |
| 19:51:09 | melwitt | efried, mriedem: working on this old service uuid upgrade bug, IIUC because of this change https://review.openstack.org/620711 if we do a service delete followed by a service create, the running compute service will never create the resource provider and compute node again, for the same 'host', because it knows about it already in the RT | |
| 19:51:39 | melwitt | even after another update_available_resource interval | |
| 19:52:07 | efried | hm, I thought someone fixed that recently. | |
| 19:52:19 | melwitt | the func tests (written before that change landed) now fail because can't get the RP created again | |
| 19:53:16 | efried | oh, this is if you delete and recreate the service without actually stopping the running process? | |
| 19:53:17 | melwitt | I rebased the set today, so I should have all of the latest changes | |
| 19:53:22 | melwitt | correct | |
| 19:53:40 | efried | Yes, this was definitely addressed recently, but I thought someone killed the change for reasons. Let me find it... | |
| 19:53:54 | mriedem | https://review.openstack.org/#/c/641899/ ? | |
| 19:54:46 | efried | Yup, that's the one. melwitt ^ | |
| 19:54:50 | melwitt | looking | |
| 19:55:42 | melwitt | I'm not even getting to that code though, it looks like that would at least try to create the provider | |
| 19:55:53 | efried | melwitt: Does that have the wrong bug number associated with it? | |
| 19:56:08 | efried | (I'm asking, why didn't you find it if you were looking into this already?) | |
| 19:56:32 | efried | duplicate bugs? | |
| 19:56:46 | melwitt | oh they're saying after a service restart the RP create fails | |
| 19:57:03 | melwitt | efried: oh sorry, I'm working on refreshing patches for an unrelated bug | |
| 19:57:30 | melwitt | and ran into this problem of the RP will never be created again while the service is running | |
| 19:57:50 | melwitt | because the func tests I'm working with are now failing because of it | |
| 19:58:27 | efried | I mean, it makes sense, because we have the provider tree cache, and no reason to think it has expired, so we never go looking for the provider to notice it's gone so we never create it. | |
| 19:58:37 | efried | This is what SIGHUP was supposed to be for. | |
| 19:59:06 | melwitt | ok, so it's a known thing that if an operator deletes and creates a service with the same hostname, they're supposed to SIGHUP now | |
| 19:59:07 | efried | And this would be a doc issue: "If you delete and recreate the service, SIGHUP (or restart ffs) your n-cpu process" | |
| 19:59:52 | efried | Well, I'm saying if it's not documented as such, that would be my first proposed solution. I'm not positive it's going to fix the problem (even when SIGHUP works, which it still doesn't afaik) but it'd be the first thing I'd try. | |
| 19:59:58 | mriedem | fwiw we already have a note about that in the API reference https://developer.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service | |
| 20:00:02 | melwitt | ok. I can simulate that in the test, but wanted to mention it in case it wasn't desired/expected | |
| 20:00:20 | melwitt | thanks | |
| 20:00:53 | efried | cool, the doc actually says to stop the thing, which makes freakin sense to me. | |
| 20:01:11 | efried | Maybe I'm being too simplistic in my thinking | |
| 20:01:23 | efried | but how does an operator expect "delete a thing but keep it running" to ever work? | |
| 20:01:36 | melwitt | no, it's fine, I just saw a change and didn't have the context about it | |
| 20:02:14 | melwitt | well, the delete it and create it again, so they expect it to be running after they create it, in this scenario | |
| 20:02:14 | efried | sorry, tbc I'm not griping at you; I guess I'm kind of wtf-ing that there's a bug about this. | |
| 20:02:29 | efried | Perhaps we should have delete kill the service | |
| 20:02:39 | efried | I think that may be what cdent was suggesting in the above patch | |
| 20:03:05 | melwitt | I can't remember why operators have done that, something to do with an upgrade. cfriesen ran into it before I think | |
| 20:03:36 | efried | ...or make the delete operation fail if the service is running, to force the operator to do it. | |
| 20:03:42 | melwitt | (delete service, create service, for same hostname) | |
| 20:04:03 | efried | Oh, I'm *sure* there's a good reason to do that ^ | |
| 20:04:05 | efried | :) | |
| 20:04:34 | efried | but expecting the running process to survive intact and work properly through it... | |
| 20:05:09 | efried | perhaps it used to work by pure luck and now they're facing RBB. | |
| 20:06:07 | cfriesen | I think we probably hit it during upgrade...our upgrade mechanism was to delete the standby controller, then reinstall it with the new load (and the same hostname as before), then migrate the DB over and cut over to the new controller. | |
| 20:07:11 | melwitt | cfriesen: that sounds like what I recall | |
| 20:07:48 | cfriesen | hmm..that wouldn't delete the nova-compute service though. | |
| 20:08:36 | melwitt | cfriesen: yeah this was deleting the nova-compute service for a host, then recreating it. deleting the nova-compute service record would kill the compute node record too | |
| 20:08:53 | melwitt | and you're saying your upgrade routine would not do that | |
| 20:10:43 | cfriesen | I don't think so...trying to find the context for https://bugs.launchpad.net/nova/+bug/1764556 | |
| 20:10:45 | openstack | Launchpad bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 20:11:25 | melwitt | cfriesen: it's ok, don't spend time on it. we were just talking about the whole service delete and create thing. behavior changed semi recently | |
| 20:11:42 | melwitt | and I thought I remembered you might have done something like that in the past | |
| 20:11:52 | cfriesen | okay, so it looks like for that bug we *did* delete a compute node while on Newton, then migrated to Pike, then created a new node with the same name. May have been an artificial test though, not actually required for upgrade. | |
| 20:12:06 | cfriesen | upgraded to Pike, rather | |
| 20:12:12 | melwitt | gotcha | |
| 20:16:13 | mriedem | the delete compute service while it's still running was never really thought out i don't think | |
| 20:16:23 | mriedem | we've only semi recently tried to shore up a lot of that nonsense | |
| 20:16:47 | mriedem | it's only been noticed as more of a problem since we started building onto it with things like host mappings and resource providers | |
| 20:17:33 | mriedem | i.e. this is semi recent https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L247 | |
| 20:17:41 | mriedem | and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L270 | |
| 20:17:46 | mriedem | and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L275 | |