Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-10
16:47:45 mriedem dansmith: just finished
16:48:02 mlavalle mriedem: excatly and sisnce I have only one compute, that was the end of that request
16:48:05 dansmith okay gerrit didn't notify me
16:48:44 mriedem mlavalle: did enabling debug in the scheduler and checking the logs tell you what was up? or did you have to then look at the code too?
16:49:18 stephenfin mriedem: fwiw, your assumption that the issue I pointed out earlier wasn't ironic specific seems to have proven out. I spotted this after http://paste.openstack.org/show/fsWdgjpwsycPzdqPXNLI/
16:49:46 stephenfin Not sure why that's happening (I suspect something HA'y) but that would explain why ID, an auto-incremented primary key field (I assume) wasn't set
16:50:41 stephenfin I found a similar issue reported by MarkusZ years ago that I need to look into more, but that's tomorrow's work https://bugs.launchpad.net/nova/+bug/1566783/comments/4
16:50:43 openstack Launchpad bug 1566783 in OpenStack Compute (nova) "On service start, nova-compute fails to register itself to the DB" [Low,Confirmed]
16:50:45 mriedem stephenfin: that's a failure to create the compute node in the db because some other node with the same uuid already exists, but you wouldn't have a compute_nodes table record without an id
16:50:47 stephenfin Just closing that loop :)
16:50:59 mriedem so i still don't understand how you'd get that object action error on loading ComputeNode.id
16:51:10 mlavalle mriedem: without enabling debug, the conductor log gave the filter that was failing. at the end the conductor logs how many hosts each filter returns. All the filters looked something like: in: 1, out 1. The image filter was: in: 1, out: 0
16:51:45 stephenfin mriedem: I figured the initial object creation was failing so we're left with a Python object but no underlying DB entry
16:51:48 mriedem mlavalle: sure, just wondering if it was really hard to figure out but sounds like it wasn't
16:52:02 mriedem stephenfin: we shouldn't even get to _setup_pci_tracker in that case though
16:52:34 mlavalle mriedem: nop, just looking closer at the logs. I guess when I pinged you, it was Friday, night and I was tired..... Or maybe I just needed an excuse to say hi ;-)
16:53:10 mriedem stephenfin: i.e. if this fails https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L584 you're not going to be calling https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L589
16:53:34 mriedem the question would be why would we even try to create the compute node record if it already exists by uuid
16:53:59 mriedem because https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L567 should find the existing cn in the db
16:54:14 mriedem unless,
16:54:19 mriedem the hypervisor_hostname is different
16:54:47 stephenfin Could it be racy if you've multiple compute services running?
16:54:51 sean-k-mooney i was going to say we lookup the compute service by hostname to get the uuid
16:55:43 mriedem it could race if you've got multiple compute services trying to manage the same node yeah
16:56:19 sean-k-mooney or multiple nodes with the same hostname?
16:56:23 mriedem stephenfin: do you have this fix? https://github.com/openstack/nova/commit/57566f4c8d2929c25e76564883369a7c6eda720a
16:57:06 mriedem sean-k-mooney: i'm not sure what you're saying
16:57:16 mriedem "compute service by hostname to get the uuid" is wrong though in this case
16:57:36 mriedem compute nodes in the db are 1:1 with ironic nodes, and in this case we're looking up the compute node by nodename, i.e. hypervisor_hostname, i.e. ironic node.uuid
16:57:42 mriedem not the nova-compute service hostname
16:57:54 sean-k-mooney oh this is ironic never mind
16:58:03 stephenfin Damn it, I've to go (tag rugby match in 15 minutes). Sorry /o\
16:58:14 mriedem back to the pile
16:58:21 stephenfin yuuuup
16:59:12 sean-k-mooney i was going to say if there are two hosts with the same hostname in the cloud then the both endup sharing the same compute service and hypervior entry in the db and evertying is broken.
16:59:26 sean-k-mooney thats not related to that however
17:00:08 cfriesen off-the-wall question here...with the rise of containerized openstack services it's a bit of a pain to tweak the code since killing the running service kills the container. has anyone considered a way to "hot reload" new code without killing the running nova process?
17:00:54 sean-k-mooney cfriesen: as in via a SIG_HUP
17:01:11 sean-k-mooney i dont think so
17:01:22 sean-k-mooney whats the issue with restarting the container
17:01:57 cfriesen sean-k-mooney: you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image...
17:02:03 cdent or starting a different container to take over
17:02:19 sean-k-mooney cdent: oh i taught you meet jsut execing into the contanter and patching it
17:02:27 sean-k-mooney cfriesen: ^
17:02:27 cfriesen sean-k-mooney: that's what I'd like to do
17:02:49 cfriesen sean-k-mooney: but if you stop nova-compute (for example) the container dies and a new unpatched one starts up
17:02:50 cdent but the nova process is the only one
17:02:50 sean-k-mooney cfriesen: in that case build your continer form git repos instad of tarballs
17:03:31 sean-k-mooney cfriesen: that depends on how you run the container
17:03:55 cdent cfriesen: it's not impossible to reload python code in the same process, but it is very icky
17:03:59 sean-k-mooney your change inside a contaienr dont disappare on a docker reaster nova_compute
17:04:20 sean-k-mooney cdent: and eventlet monkeypatching makes it even harder
17:04:24 cdent yes
17:04:26 jangutter cfriesen: docker containers ? I've hotpatched them, done 'docker commit' and 'docker restart'.
17:04:53 jangutter cfriesen: ugly, but hey, so am I.
17:05:04 sean-k-mooney jangutter: or docker exec then edit it and docker restart
17:05:22 jangutter sean-k-mooney: yep, that too.
17:05:30 cfriesen jangutter: do you know how that behaves when the container is part of a pod that is part of a deployment or statefulset?
17:05:50 sean-k-mooney cfriesen: it works fine in a pod
17:06:14 jangutter cfriesen: does the answer involve a singularity that slowly consumes all in its path?
17:06:18 sean-k-mooney you are not deleting the contiaer so the deployment does not recreated the pod
17:07:03 cfriesen sounds promising...will have to try that.
17:07:48 sean-k-mooney cfriesen: but modifying docker conatiner is generally considerd heresy. in k8s land its even more vilified
17:07:50 jangutter cfriesen: protip, pass '-u root' when doing the exec since you don't always have permissions.
17:08:40 jangutter cfriesen, sean-k-mooney: yeah, there's no guarantee any of the changes will persist or not. at best it's just something you do before you destroy the system.
17:08:45 sean-k-mooney cfriesen: if you were using kolla-ansible they have a dev mode where the clone the git repo on the host and just bind mont into the container i think
17:09:37 dansmith mriedem: am I missing what bug this actually fixes? https://review.openstack.org/#/c/647713/1
17:09:44 sean-k-mooney the intended worflow being edit on host then docker restart nova_compute
17:10:01 cfriesen sean-k-mooney: yeah, that'd be cleaner
17:12:50 cdent cfriesen: is this sort of live code tweaking a thing you find common?
17:13:10 cdent I mean I can imagine it often enough in test settings, experiments, and the like
17:13:37 cdent but in a deployed world aren't the choices usually: observe, kill, make a new one?
17:14:09 mnaser there's not really any pythonic api to manage cells, right?
17:14:14 mnaser or any api at all
17:15:37 mnaser I am looking at adding multicell support in openstackansible, one of the things to introduce is the addition of some way to define cells
17:15:50 cfriesen cdent: this would be for tweaking stuff on lab systems. I agree for areas where devstack is sufficient that'd be better.
17:18:14 cdent cfriesen: I guess I was thinking a lab system is one of the main areas where "you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image..." wouldn't be too much of a big deal, and you might have some simple(-ish) tooling to support it
17:18:27 cdent I agree it is a PITA
17:18:43 dansmith mnaser: you mean via the external API I assume? there isn't by design
17:19:08 dansmith mnaser: or do you mean something like being able to import nova-manage and use its routines?
17:19:15 mnaser dansmith: yeah.. or I was thinking maybe we could do something like.. import nova.cmd.manage; manage.add_cell() or whatever
17:19:21 cfriesen cdent: tool improvements would help, I agree.
17:19:26 mnaser #2 (because ansible can just have python scripts run)
17:19:30 dansmith mnaser: yeah, that could be a thing, but isn't currently
17:19:49 mnaser fair enough, so for now probably best to stick to parsing CLI output
17:20:25 dansmith mnaser: tbh, I think you probably don't want to import and run in your own namespace, since talking to cells requires pivoting global oslo.db state and other things
17:21:38 mnaser dansmith: fair enough.. so probably better off just doing CLI operations
17:21:56 dansmith mnaser: the cli commands aim to be pretty idempotent, but if we need --robot output formatting or something, then we should do that
17:26:33 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Use update_provider_tree in vmware virt driver https://review.openstack.org/651615
17:45:36 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Nova LLC allocation - RMD plugin for RDT CAT https://review.openstack.org/651233
17:56:12 mriedem dansmith: it's a perf improvement to avoid a call to get allocations for the instance created by the scheduler when we already have those in the Selection object in scope
17:56:29 dansmith mriedem: ..right I got that ;)
17:56:48 dansmith is that really substantial enough to backport?
17:58:44 mriedem it's just new code in stein and missed the cutoff so it's not controversial imo
17:58:49 mriedem we wouldn't die without it no
17:59:21 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Nova local resource management that uses RMD https://review.openstack.org/651130
18:57:32 openstackgerrit Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Base Enablement https://review.openstack.org/651130
19:06:32 mriedem melwitt: i re-read the counting quotas from placement spec for train again and noticed a few things that could be cleaned up https://review.openstack.org/#/c/645302/
19:06:59 melwitt mriedem: ok, thanks
19:07:40 melwitt mriedem: to your other comment (I'll reply on the review too), we're not counting unmigrated qfd instance mappings because we're falling back to legacy counting if unmigrated qfd instance mappings are detected

Earlier   Later