Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-10
16:22:27 bauzas melwitt: thanks for reviewing https://review.openstack.org/#/c/650963/ I replied to your question
16:23:24 melwitt bauzas: a-ha, thanks
16:30:00 melwitt lyarwood, mdbooth: begging for reviews on this again, disk unit stuff https://review.openstack.org/611974
16:30:37 lyarwood melwitt: ack saw above, I'll take a look once I've finished with all of this downstream paperwork... bugzilla--
16:30:49 melwitt oh cool, sorry for the double ping
16:31:01 lyarwood np at all
16:34:28 mriedem efried_rollin: want to drop the -W on https://review.openstack.org/#/c/649600/ ?
16:35:11 openstackgerrit Theodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.openstack.org/648686
16:37:37 openstackgerrit melanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556 https://review.openstack.org/562041
16:37:39 openstack bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1778305 https://review.openstack.org/582407
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Don't generate service UUID for deleted services https://review.openstack.org/582408
16:37:40 openstack bug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Undecided,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)
16:46:00 mriedem mlavalle: did you sort out the NoValidHost error you were getting the other night?
16:46:38 mlavalle mriedem: yes, thanks for following up. it was the image filter
16:47:34 mriedem mlavalle: ok so you have some image with metadata that didn't match any hosts i'm guessing?
16:47:40 dansmith mriedem: were you going to un-WIP your stein backports?
16:47:45 mriedem dansmith: just finished
16:48:02 mlavalle mriedem: excatly and sisnce I have only one compute, that was the end of that request
16:48:05 dansmith okay gerrit didn't notify me
16:48:44 mriedem mlavalle: did enabling debug in the scheduler and checking the logs tell you what was up? or did you have to then look at the code too?
16:49:18 stephenfin mriedem: fwiw, your assumption that the issue I pointed out earlier wasn't ironic specific seems to have proven out. I spotted this after http://paste.openstack.org/show/fsWdgjpwsycPzdqPXNLI/
16:49:46 stephenfin Not sure why that's happening (I suspect something HA'y) but that would explain why ID, an auto-incremented primary key field (I assume) wasn't set
16:50:41 stephenfin I found a similar issue reported by MarkusZ years ago that I need to look into more, but that's tomorrow's work https://bugs.launchpad.net/nova/+bug/1566783/comments/4
16:50:43 openstack Launchpad bug 1566783 in OpenStack Compute (nova) "On service start, nova-compute fails to register itself to the DB" [Low,Confirmed]
16:50:45 mriedem stephenfin: that's a failure to create the compute node in the db because some other node with the same uuid already exists, but you wouldn't have a compute_nodes table record without an id
16:50:47 stephenfin Just closing that loop :)
16:50:59 mriedem so i still don't understand how you'd get that object action error on loading ComputeNode.id
16:51:10 mlavalle mriedem: without enabling debug, the conductor log gave the filter that was failing. at the end the conductor logs how many hosts each filter returns. All the filters looked something like: in: 1, out 1. The image filter was: in: 1, out: 0
16:51:45 stephenfin mriedem: I figured the initial object creation was failing so we're left with a Python object but no underlying DB entry
16:51:48 mriedem mlavalle: sure, just wondering if it was really hard to figure out but sounds like it wasn't
16:52:02 mriedem stephenfin: we shouldn't even get to _setup_pci_tracker in that case though
16:52:34 mlavalle mriedem: nop, just looking closer at the logs. I guess when I pinged you, it was Friday, night and I was tired..... Or maybe I just needed an excuse to say hi ;-)
16:53:10 mriedem stephenfin: i.e. if this fails https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L584 you're not going to be calling https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L589
16:53:34 mriedem the question would be why would we even try to create the compute node record if it already exists by uuid
16:53:59 mriedem because https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L567 should find the existing cn in the db
16:54:14 mriedem unless,
16:54:19 mriedem the hypervisor_hostname is different
16:54:47 stephenfin Could it be racy if you've multiple compute services running?
16:54:51 sean-k-mooney i was going to say we lookup the compute service by hostname to get the uuid
16:55:43 mriedem it could race if you've got multiple compute services trying to manage the same node yeah
16:56:19 sean-k-mooney or multiple nodes with the same hostname?
16:56:23 mriedem stephenfin: do you have this fix? https://github.com/openstack/nova/commit/57566f4c8d2929c25e76564883369a7c6eda720a
16:57:06 mriedem sean-k-mooney: i'm not sure what you're saying
16:57:16 mriedem "compute service by hostname to get the uuid" is wrong though in this case
16:57:36 mriedem compute nodes in the db are 1:1 with ironic nodes, and in this case we're looking up the compute node by nodename, i.e. hypervisor_hostname, i.e. ironic node.uuid
16:57:42 mriedem not the nova-compute service hostname
16:57:54 sean-k-mooney oh this is ironic never mind
16:58:03 stephenfin Damn it, I've to go (tag rugby match in 15 minutes). Sorry /o\
16:58:14 mriedem back to the pile
16:58:21 stephenfin yuuuup
16:59:12 sean-k-mooney i was going to say if there are two hosts with the same hostname in the cloud then the both endup sharing the same compute service and hypervior entry in the db and evertying is broken.
16:59:26 sean-k-mooney thats not related to that however
17:00:08 cfriesen off-the-wall question here...with the rise of containerized openstack services it's a bit of a pain to tweak the code since killing the running service kills the container. has anyone considered a way to "hot reload" new code without killing the running nova process?
17:00:54 sean-k-mooney cfriesen: as in via a SIG_HUP
17:01:11 sean-k-mooney i dont think so
17:01:22 sean-k-mooney whats the issue with restarting the container
17:01:57 cfriesen sean-k-mooney: you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image...
17:02:03 cdent or starting a different container to take over
17:02:19 sean-k-mooney cdent: oh i taught you meet jsut execing into the contanter and patching it
17:02:27 sean-k-mooney cfriesen: ^
17:02:27 cfriesen sean-k-mooney: that's what I'd like to do
17:02:49 cfriesen sean-k-mooney: but if you stop nova-compute (for example) the container dies and a new unpatched one starts up
17:02:50 cdent but the nova process is the only one
17:02:50 sean-k-mooney cfriesen: in that case build your continer form git repos instad of tarballs
17:03:31 sean-k-mooney cfriesen: that depends on how you run the container
17:03:55 cdent cfriesen: it's not impossible to reload python code in the same process, but it is very icky
17:03:59 sean-k-mooney your change inside a contaienr dont disappare on a docker reaster nova_compute
17:04:20 sean-k-mooney cdent: and eventlet monkeypatching makes it even harder
17:04:24 cdent yes
17:04:26 jangutter cfriesen: docker containers ? I've hotpatched them, done 'docker commit' and 'docker restart'.
17:04:53 jangutter cfriesen: ugly, but hey, so am I.
17:05:04 sean-k-mooney jangutter: or docker exec then edit it and docker restart
17:05:22 jangutter sean-k-mooney: yep, that too.
17:05:30 cfriesen jangutter: do you know how that behaves when the container is part of a pod that is part of a deployment or statefulset?
17:05:50 sean-k-mooney cfriesen: it works fine in a pod
17:06:14 jangutter cfriesen: does the answer involve a singularity that slowly consumes all in its path?
17:06:18 sean-k-mooney you are not deleting the contiaer so the deployment does not recreated the pod
17:07:03 cfriesen sounds promising...will have to try that.
17:07:48 sean-k-mooney cfriesen: but modifying docker conatiner is generally considerd heresy. in k8s land its even more vilified
17:07:50 jangutter cfriesen: protip, pass '-u root' when doing the exec since you don't always have permissions.
17:08:40 jangutter cfriesen, sean-k-mooney: yeah, there's no guarantee any of the changes will persist or not. at best it's just something you do before you destroy the system.
17:08:45 sean-k-mooney cfriesen: if you were using kolla-ansible they have a dev mode where the clone the git repo on the host and just bind mont into the container i think
17:09:37 dansmith mriedem: am I missing what bug this actually fixes? https://review.openstack.org/#/c/647713/1
17:09:44 sean-k-mooney the intended worflow being edit on host then docker restart nova_compute
17:10:01 cfriesen sean-k-mooney: yeah, that'd be cleaner
17:12:50 cdent cfriesen: is this sort of live code tweaking a thing you find common?
17:13:10 cdent I mean I can imagine it often enough in test settings, experiments, and the like
17:13:37 cdent but in a deployed world aren't the choices usually: observe, kill, make a new one?
17:14:09 mnaser there's not really any pythonic api to manage cells, right?
17:14:14 mnaser or any api at all
17:15:37 mnaser I am looking at adding multicell support in openstackansible, one of the things to introduce is the addition of some way to define cells
17:15:50 cfriesen cdent: this would be for tweaking stuff on lab systems. I agree for areas where devstack is sufficient that'd be better.
17:18:14 cdent cfriesen: I guess I was thinking a lab system is one of the main areas where "you have to make a new image, then tag it, and upload it to the repository, then tell helm to use the new image..." wouldn't be too much of a big deal, and you might have some simple(-ish) tooling to support it
17:18:27 cdent I agree it is a PITA
17:18:43 dansmith mnaser: you mean via the external API I assume? there isn't by design
17:19:08 dansmith mnaser: or do you mean something like being able to import nova-manage and use its routines?
17:19:15 mnaser dansmith: yeah.. or I was thinking maybe we could do something like.. import nova.cmd.manage; manage.add_cell() or whatever
17:19:21 cfriesen cdent: tool improvements would help, I agree.
17:19:26 mnaser #2 (because ansible can just have python scripts run)

Earlier   Later