| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-10 | |||
| 16:59:26 | sean-k-mooney | thats not related to that however | |
| 17:00:08 | cfriesen | off-the-wall question here...with the rise of containerized openstack services it's a bit of a pain to tweak the code since killing the running service kills the container. has anyone considered a way to "hot reload" new code without killing the running nova process? | |
| 17:00:54 | sean-k-mooney | cfriesen: as in via a SIG_HUP | |
| 17:01:11 | sean-k-mooney | i dont think so | |
| 17:01:22 | sean-k-mooney | whats the issue with restarting the container | |
| 17:01:57 | cfriesen | sean-k-mooney: you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image... | |
| 17:02:03 | cdent | or starting a different container to take over | |
| 17:02:19 | sean-k-mooney | cdent: oh i taught you meet jsut execing into the contanter and patching it | |
| 17:02:27 | cfriesen | sean-k-mooney: that's what I'd like to do | |
| 17:02:27 | sean-k-mooney | cfriesen: ^ | |
| 17:02:49 | cfriesen | sean-k-mooney: but if you stop nova-compute (for example) the container dies and a new unpatched one starts up | |
| 17:02:50 | sean-k-mooney | cfriesen: in that case build your continer form git repos instad of tarballs | |
| 17:02:50 | cdent | but the nova process is the only one | |
| 17:03:31 | sean-k-mooney | cfriesen: that depends on how you run the container | |
| 17:03:55 | cdent | cfriesen: it's not impossible to reload python code in the same process, but it is very icky | |
| 17:03:59 | sean-k-mooney | your change inside a contaienr dont disappare on a docker reaster nova_compute | |
| 17:04:20 | sean-k-mooney | cdent: and eventlet monkeypatching makes it even harder | |
| 17:04:24 | cdent | yes | |
| 17:04:26 | jangutter | cfriesen: docker containers ? I've hotpatched them, done 'docker commit' and 'docker restart'. | |
| 17:04:53 | jangutter | cfriesen: ugly, but hey, so am I. | |
| 17:05:04 | sean-k-mooney | jangutter: or docker exec then edit it and docker restart | |
| 17:05:22 | jangutter | sean-k-mooney: yep, that too. | |
| 17:05:30 | cfriesen | jangutter: do you know how that behaves when the container is part of a pod that is part of a deployment or statefulset? | |
| 17:05:50 | sean-k-mooney | cfriesen: it works fine in a pod | |
| 17:06:14 | jangutter | cfriesen: does the answer involve a singularity that slowly consumes all in its path? | |
| 17:06:18 | sean-k-mooney | you are not deleting the contiaer so the deployment does not recreated the pod | |
| 17:07:03 | cfriesen | sounds promising...will have to try that. | |
| 17:07:48 | sean-k-mooney | cfriesen: but modifying docker conatiner is generally considerd heresy. in k8s land its even more vilified | |
| 17:07:50 | jangutter | cfriesen: protip, pass '-u root' when doing the exec since you don't always have permissions. | |
| 17:08:40 | jangutter | cfriesen, sean-k-mooney: yeah, there's no guarantee any of the changes will persist or not. at best it's just something you do before you destroy the system. | |
| 17:08:45 | sean-k-mooney | cfriesen: if you were using kolla-ansible they have a dev mode where the clone the git repo on the host and just bind mont into the container i think | |
| 17:09:37 | dansmith | mriedem: am I missing what bug this actually fixes? https://review.openstack.org/#/c/647713/1 | |
| 17:09:44 | sean-k-mooney | the intended worflow being edit on host then docker restart nova_compute | |
| 17:10:01 | cfriesen | sean-k-mooney: yeah, that'd be cleaner | |
| 17:12:50 | cdent | cfriesen: is this sort of live code tweaking a thing you find common? | |
| 17:13:10 | cdent | I mean I can imagine it often enough in test settings, experiments, and the like | |
| 17:13:37 | cdent | but in a deployed world aren't the choices usually: observe, kill, make a new one? | |
| 17:14:09 | mnaser | there's not really any pythonic api to manage cells, right? | |
| 17:14:14 | mnaser | or any api at all | |
| 17:15:37 | mnaser | I am looking at adding multicell support in openstackansible, one of the things to introduce is the addition of some way to define cells | |
| 17:15:50 | cfriesen | cdent: this would be for tweaking stuff on lab systems. I agree for areas where devstack is sufficient that'd be better. | |
| 17:18:14 | cdent | cfriesen: I guess I was thinking a lab system is one of the main areas where "you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image..." wouldn't be too much of a big deal, and you might have some simple(-ish) tooling to support it | |
| 17:18:27 | cdent | I agree it is a PITA | |
| 17:18:43 | dansmith | mnaser: you mean via the external API I assume? there isn't by design | |
| 17:19:08 | dansmith | mnaser: or do you mean something like being able to import nova-manage and use its routines? | |
| 17:19:15 | mnaser | dansmith: yeah.. or I was thinking maybe we could do something like.. import nova.cmd.manage; manage.add_cell() or whatever | |
| 17:19:21 | cfriesen | cdent: tool improvements would help, I agree. | |
| 17:19:26 | mnaser | #2 (because ansible can just have python scripts run) | |
| 17:19:30 | dansmith | mnaser: yeah, that could be a thing, but isn't currently | |
| 17:19:49 | mnaser | fair enough, so for now probably best to stick to parsing CLI output | |
| 17:20:25 | dansmith | mnaser: tbh, I think you probably don't want to import and run in your own namespace, since talking to cells requires pivoting global oslo.db state and other things | |
| 17:21:38 | mnaser | dansmith: fair enough.. so probably better off just doing CLI operations | |
| 17:21:56 | dansmith | mnaser: the cli commands aim to be pretty idempotent, but if we need --robot output formatting or something, then we should do that | |
| 17:26:33 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use update_provider_tree in vmware virt driver https://review.openstack.org/651615 | |
| 17:45:36 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Nova LLC allocation - RMD plugin for RDT CAT https://review.openstack.org/651233 | |
| 17:56:12 | mriedem | dansmith: it's a perf improvement to avoid a call to get allocations for the instance created by the scheduler when we already have those in the Selection object in scope | |
| 17:56:29 | dansmith | mriedem: ..right I got that ;) | |
| 17:56:48 | dansmith | is that really substantial enough to backport? | |
| 17:58:44 | mriedem | it's just new code in stein and missed the cutoff so it's not controversial imo | |
| 17:58:49 | mriedem | we wouldn't die without it no | |
| 17:59:21 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Nova local resource management that uses RMD https://review.openstack.org/651130 | |
| 18:57:32 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Base Enablement https://review.openstack.org/651130 | |
| 19:06:32 | mriedem | melwitt: i re-read the counting quotas from placement spec for train again and noticed a few things that could be cleaned up https://review.openstack.org/#/c/645302/ | |
| 19:06:59 | melwitt | mriedem: ok, thanks | |
| 19:07:40 | melwitt | mriedem: to your other comment (I'll reply on the review too), we're not counting unmigrated qfd instance mappings because we're falling back to legacy counting if unmigrated qfd instance mappings are detected | |
| 19:08:06 | melwitt | (as opposed to counting unmigrated and being potentially wrong in the count) | |
| 19:09:52 | mriedem | right i gathered that from the exchange between you and surya but i hadn't looked at the code yet that does the fallback logic | |
| 19:12:14 | melwitt | oh, I see | |
| 19:12:51 | melwitt | I misunderstood what you meant by saying you needed to load the context of how it will be used | |
| 19:20:04 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Last Level Cache https://review.openstack.org/651233 | |
| 19:37:31 | efried_rollin | mriedem: done | |
| 19:51:09 | melwitt | efried, mriedem: working on this old service uuid upgrade bug, IIUC because of this change https://review.openstack.org/620711 if we do a service delete followed by a service create, the running compute service will never create the resource provider and compute node again, for the same 'host', because it knows about it already in the RT | |
| 19:51:39 | melwitt | even after another update_available_resource interval | |
| 19:52:07 | efried | hm, I thought someone fixed that recently. | |
| 19:52:19 | melwitt | the func tests (written before that change landed) now fail because can't get the RP created again | |
| 19:53:16 | efried | oh, this is if you delete and recreate the service without actually stopping the running process? | |
| 19:53:17 | melwitt | I rebased the set today, so I should have all of the latest changes | |
| 19:53:22 | melwitt | correct | |
| 19:53:40 | efried | Yes, this was definitely addressed recently, but I thought someone killed the change for reasons. Let me find it... | |
| 19:53:54 | mriedem | https://review.openstack.org/#/c/641899/ ? | |
| 19:54:46 | efried | Yup, that's the one. melwitt ^ | |
| 19:54:50 | melwitt | looking | |
| 19:55:42 | melwitt | I'm not even getting to that code though, it looks like that would at least try to create the provider | |
| 19:55:53 | efried | melwitt: Does that have the wrong bug number associated with it? | |
| 19:56:08 | efried | (I'm asking, why didn't you find it if you were looking into this already?) | |
| 19:56:32 | efried | duplicate bugs? | |
| 19:56:46 | melwitt | oh they're saying after a service restart the RP create fails | |
| 19:57:03 | melwitt | efried: oh sorry, I'm working on refreshing patches for an unrelated bug | |
| 19:57:30 | melwitt | and ran into this problem of the RP will never be created again while the service is running | |
| 19:57:50 | melwitt | because the func tests I'm working with are now failing because of it | |
| 19:58:27 | efried | I mean, it makes sense, because we have the provider tree cache, and no reason to think it has expired, so we never go looking for the provider to notice it's gone so we never create it. | |
| 19:58:37 | efried | This is what SIGHUP was supposed to be for. | |
| 19:59:06 | melwitt | ok, so it's a known thing that if an operator deletes and creates a service with the same hostname, they're supposed to SIGHUP now | |
| 19:59:07 | efried | And this would be a doc issue: "If you delete and recreate the service, SIGHUP (or restart ffs) your n-cpu process" | |
| 19:59:52 | efried | Well, I'm saying if it's not documented as such, that would be my first proposed solution. I'm not positive it's going to fix the problem (even when SIGHUP works, which it still doesn't afaik) but it'd be the first thing I'd try. | |
| 19:59:58 | mriedem | fwiw we already have a note about that in the API reference https://developer.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service | |
| 20:00:02 | melwitt | ok. I can simulate that in the test, but wanted to mention it in case it wasn't desired/expected | |
| 20:00:20 | melwitt | thanks | |
| 20:00:53 | efried | cool, the doc actually says to stop the thing, which makes freakin sense to me. | |
| 20:01:11 | efried | Maybe I'm being too simplistic in my thinking | |