Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-10
15:35:12 cdent i imagine so
15:35:20 cdent what was the original purpose of burying?
15:35:22 dansmith yeah that's not okay
15:35:27 melwitt there's this one too but people were meh about reviewing it too https://review.openstack.org/582408
15:35:27 mriedem because it's more like (1) only bury if using a microversion to allow unburying except (2) unless you didn't request personality files etc
15:36:00 mriedem cdent: need to be able to show the servers from the API even though they are in error state and not on a host,
15:36:01 dansmith cdent: it's still the original purpose, which is that if we didn't pick a host, we didn't pick a cell, thus we have nowhere to store it
15:36:12 mriedem and the api pulls servers from a cell db
15:36:39 mriedem we have build_requests but those are short-lived
15:36:44 cdent ah, right, I keep forgetting that a server that is an ERROR is still a server (which is a decision I've never quite understood)
15:37:14 mriedem melwitt: it's failing pep8 :)
15:37:38 bauzas mriedem: ack, sorry on a meeting
15:38:43 melwitt sigh
15:40:52 openstackgerrit Surya Seetharaman proposed openstack/nova master: Block automatic transport_url update for cell0 https://review.openstack.org/605414
15:46:38 openstackgerrit Merged openstack/nova-specs master: Re-propose cross-cell-resize spec for Train https://review.openstack.org/642807
16:22:27 bauzas melwitt: thanks for reviewing https://review.openstack.org/#/c/650963/ I replied to your question
16:23:24 melwitt bauzas: a-ha, thanks
16:30:00 melwitt lyarwood, mdbooth: begging for reviews on this again, disk unit stuff https://review.openstack.org/611974
16:30:37 lyarwood melwitt: ack saw above, I'll take a look once I've finished with all of this downstream paperwork... bugzilla--
16:30:49 melwitt oh cool, sorry for the double ping
16:31:01 lyarwood np at all
16:34:28 mriedem efried_rollin: want to drop the -W on https://review.openstack.org/#/c/649600/ ?
16:35:11 openstackgerrit Theodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.openstack.org/648686
16:37:37 openstackgerrit melanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556 https://review.openstack.org/562041
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Don't generate service UUID for deleted services https://review.openstack.org/582408
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1778305 https://review.openstack.org/582407
16:37:39 openstack bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)
16:37:40 openstack bug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Undecided,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)
16:46:00 mriedem mlavalle: did you sort out the NoValidHost error you were getting the other night?
16:46:38 mlavalle mriedem: yes, thanks for following up. it was the image filter
16:47:34 mriedem mlavalle: ok so you have some image with metadata that didn't match any hosts i'm guessing?
16:47:40 dansmith mriedem: were you going to un-WIP your stein backports?
16:47:45 mriedem dansmith: just finished
16:48:02 mlavalle mriedem: excatly and sisnce I have only one compute, that was the end of that request
16:48:05 dansmith okay gerrit didn't notify me
16:48:44 mriedem mlavalle: did enabling debug in the scheduler and checking the logs tell you what was up? or did you have to then look at the code too?
16:49:18 stephenfin mriedem: fwiw, your assumption that the issue I pointed out earlier wasn't ironic specific seems to have proven out. I spotted this after http://paste.openstack.org/show/fsWdgjpwsycPzdqPXNLI/
16:49:46 stephenfin Not sure why that's happening (I suspect something HA'y) but that would explain why ID, an auto-incremented primary key field (I assume) wasn't set
16:50:41 stephenfin I found a similar issue reported by MarkusZ years ago that I need to look into more, but that's tomorrow's work https://bugs.launchpad.net/nova/+bug/1566783/comments/4
16:50:43 openstack Launchpad bug 1566783 in OpenStack Compute (nova) "On service start, nova-compute fails to register itself to the DB" [Low,Confirmed]
16:50:45 mriedem stephenfin: that's a failure to create the compute node in the db because some other node with the same uuid already exists, but you wouldn't have a compute_nodes table record without an id
16:50:47 stephenfin Just closing that loop :)
16:50:59 mriedem so i still don't understand how you'd get that object action error on loading ComputeNode.id
16:51:10 mlavalle mriedem: without enabling debug, the conductor log gave the filter that was failing. at the end the conductor logs how many hosts each filter returns. All the filters looked something like: in: 1, out 1. The image filter was: in: 1, out: 0
16:51:45 stephenfin mriedem: I figured the initial object creation was failing so we're left with a Python object but no underlying DB entry
16:51:48 mriedem mlavalle: sure, just wondering if it was really hard to figure out but sounds like it wasn't
16:52:02 mriedem stephenfin: we shouldn't even get to _setup_pci_tracker in that case though
16:52:34 mlavalle mriedem: nop, just looking closer at the logs. I guess when I pinged you, it was Friday, night and I was tired..... Or maybe I just needed an excuse to say hi ;-)
16:53:10 mriedem stephenfin: i.e. if this fails https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L584 you're not going to be calling https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L589
16:53:34 mriedem the question would be why would we even try to create the compute node record if it already exists by uuid
16:53:59 mriedem because https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L567 should find the existing cn in the db
16:54:14 mriedem unless,
16:54:19 mriedem the hypervisor_hostname is different
16:54:47 stephenfin Could it be racy if you've multiple compute services running?
16:54:51 sean-k-mooney i was going to say we lookup the compute service by hostname to get the uuid
16:55:43 mriedem it could race if you've got multiple compute services trying to manage the same node yeah
16:56:19 sean-k-mooney or multiple nodes with the same hostname?
16:56:23 mriedem stephenfin: do you have this fix? https://github.com/openstack/nova/commit/57566f4c8d2929c25e76564883369a7c6eda720a
16:57:06 mriedem sean-k-mooney: i'm not sure what you're saying
16:57:16 mriedem "compute service by hostname to get the uuid" is wrong though in this case
16:57:36 mriedem compute nodes in the db are 1:1 with ironic nodes, and in this case we're looking up the compute node by nodename, i.e. hypervisor_hostname, i.e. ironic node.uuid
16:57:42 mriedem not the nova-compute service hostname
16:57:54 sean-k-mooney oh this is ironic never mind
16:58:03 stephenfin Damn it, I've to go (tag rugby match in 15 minutes). Sorry /o\
16:58:14 mriedem back to the pile
16:58:21 stephenfin yuuuup
16:59:12 sean-k-mooney i was going to say if there are two hosts with the same hostname in the cloud then the both endup sharing the same compute service and hypervior entry in the db and evertying is broken.
16:59:26 sean-k-mooney thats not related to that however
17:00:08 cfriesen off-the-wall question here...with the rise of containerized openstack services it's a bit of a pain to tweak the code since killing the running service kills the container. has anyone considered a way to "hot reload" new code without killing the running nova process?
17:00:54 sean-k-mooney cfriesen: as in via a SIG_HUP
17:01:11 sean-k-mooney i dont think so
17:01:22 sean-k-mooney whats the issue with restarting the container
17:01:57 cfriesen sean-k-mooney: you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image...
17:02:03 cdent or starting a different container to take over
17:02:19 sean-k-mooney cdent: oh i taught you meet jsut execing into the contanter and patching it
17:02:27 cfriesen sean-k-mooney: that's what I'd like to do
17:02:27 sean-k-mooney cfriesen: ^
17:02:49 cfriesen sean-k-mooney: but if you stop nova-compute (for example) the container dies and a new unpatched one starts up
17:02:50 sean-k-mooney cfriesen: in that case build your continer form git repos instad of tarballs
17:02:50 cdent but the nova process is the only one
17:03:31 sean-k-mooney cfriesen: that depends on how you run the container
17:03:55 cdent cfriesen: it's not impossible to reload python code in the same process, but it is very icky
17:03:59 sean-k-mooney your change inside a contaienr dont disappare on a docker reaster nova_compute
17:04:20 sean-k-mooney cdent: and eventlet monkeypatching makes it even harder
17:04:24 cdent yes
17:04:26 jangutter cfriesen: docker containers ? I've hotpatched them, done 'docker commit' and 'docker restart'.
17:04:53 jangutter cfriesen: ugly, but hey, so am I.
17:05:04 sean-k-mooney jangutter: or docker exec then edit it and docker restart
17:05:22 jangutter sean-k-mooney: yep, that too.
17:05:30 cfriesen jangutter: do you know how that behaves when the container is part of a pod that is part of a deployment or statefulset?
17:05:50 sean-k-mooney cfriesen: it works fine in a pod
17:06:14 jangutter cfriesen: does the answer involve a singularity that slowly consumes all in its path?
17:06:18 sean-k-mooney you are not deleting the contiaer so the deployment does not recreated the pod
17:07:03 cfriesen sounds promising...will have to try that.
17:07:48 sean-k-mooney cfriesen: but modifying docker conatiner is generally considerd heresy. in k8s land its even more vilified
17:07:50 jangutter cfriesen: protip, pass '-u root' when doing the exec since you don't always have permissions.
17:08:40 jangutter cfriesen, sean-k-mooney: yeah, there's no guarantee any of the changes will persist or not. at best it's just something you do before you destroy the system.
17:08:45 sean-k-mooney cfriesen: if you were using kolla-ansible they have a dev mode where the clone the git repo on the host and just bind mont into the container i think
17:09:37 dansmith mriedem: am I missing what bug this actually fixes? https://review.openstack.org/#/c/647713/1
17:09:44 sean-k-mooney the intended worflow being edit on host then docker restart nova_compute

Earlier   Later