Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-10
14:55:50 mriedem ok
14:57:53 mriedem stephenfin: back at you https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1823781
14:58:07 stephenfin ack
14:59:17 stephenfin mriedem: Out of curiosity, does this look familiar to you? http://paste.openstack.org/show/P6AlRRpBWnYwFfWWVJFl/ It's a stable/rocky deployment
15:00:44 mriedem yes i think so, sec
15:01:29 mriedem actually no nvm, was thinking of something else pci related
15:02:32 mriedem it's blowing up on this? "n_id = compute_node.id"
15:02:43 mriedem i only see _setup_pci_tracker called with an existing compute node record where id should be set
15:02:47 mriedem unless you guys have a fork
15:04:53 stephenfin Not that I'm aware of. I had been doing some work in that area but none of that is merged yet :/ https://review.openstack.org/#/c/649559/
15:05:54 stephenfin But yeah, looks like that. I don't know if that attribute should be set or not. I can't see why it wouldn't but all things Ironic really aren't my strong suit
15:06:20 melwitt mriedem: I was thinking dansmith would want to sign off on it
15:08:58 mriedem stephenfin: i don't think that error has anything to do with ironic
15:13:21 stephenfin Perhaps, but it's only happening with baremetal nodes so I suspect it's some with how we're initializing the Ironic driver
15:13:25 stephenfin but that's a grasp
15:15:01 mriedem so the driver returns a list of nodenames for that host and those get passed one at a time to RT https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7893
15:16:50 mriedem should eventually call through to here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L743
15:17:24 mriedem if the compute node already exists in the db you'd get here on first startup https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L567
15:18:08 mriedem this should also be ok https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L530
15:18:35 mriedem otherwise we'd create the node in the db https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L584
15:18:38 mriedem which should set the id on the cn object
15:19:07 mriedem https://github.com/openstack/nova/blob/stable/rocky/nova/objects/compute_node.py#L322
15:19:50 mriedem looks like you've got some debugging to do if you can recreate it
15:32:12 mriedem lyarwood: melwitt: dansmith: bauzas: looks like we can start approving stable/stein backports now so i'll start going through those and removing the WIP on mine
15:32:27 mriedem we need to roll through some of those to get things flushed out down through pike for pike-em
15:32:38 dansmith ack
15:32:40 melwitt ah, yep
15:33:28 melwitt I have a bug fix that needs to go back to pike but it hasn't even merged on master yet and having trouble getting review https://review.openstack.org/611974
15:34:19 cdent mriedem: does a server always get buried to cell0 when it fails to find a host? Is it possible to say "only bury if you requested a build at a microversion that will allow an unbury"?
15:34:30 mriedem melwitt: i definitely don't know that code so would need at least lyarwood and/or mdbooth to sign off
15:34:43 mriedem cdent: "does a server always get buried to cell0 when it fails to find a host?" yes
15:35:05 mriedem "Is it possible to say "only bury if you requested a build at a microversion that will allow an unbury"?" i think that gets really complicated fast
15:35:12 cdent i imagine so
15:35:20 cdent what was the original purpose of burying?
15:35:22 dansmith yeah that's not okay
15:35:27 melwitt there's this one too but people were meh about reviewing it too https://review.openstack.org/582408
15:35:27 mriedem because it's more like (1) only bury if using a microversion to allow unburying except (2) unless you didn't request personality files etc
15:36:00 mriedem cdent: need to be able to show the servers from the API even though they are in error state and not on a host,
15:36:01 dansmith cdent: it's still the original purpose, which is that if we didn't pick a host, we didn't pick a cell, thus we have nowhere to store it
15:36:12 mriedem and the api pulls servers from a cell db
15:36:39 mriedem we have build_requests but those are short-lived
15:36:44 cdent ah, right, I keep forgetting that a server that is an ERROR is still a server (which is a decision I've never quite understood)
15:37:14 mriedem melwitt: it's failing pep8 :)
15:37:38 bauzas mriedem: ack, sorry on a meeting
15:38:43 melwitt sigh
15:40:52 openstackgerrit Surya Seetharaman proposed openstack/nova master: Block automatic transport_url update for cell0 https://review.openstack.org/605414
15:46:38 openstackgerrit Merged openstack/nova-specs master: Re-propose cross-cell-resize spec for Train https://review.openstack.org/642807
16:22:27 bauzas melwitt: thanks for reviewing https://review.openstack.org/#/c/650963/ I replied to your question
16:23:24 melwitt bauzas: a-ha, thanks
16:30:00 melwitt lyarwood, mdbooth: begging for reviews on this again, disk unit stuff https://review.openstack.org/611974
16:30:37 lyarwood melwitt: ack saw above, I'll take a look once I've finished with all of this downstream paperwork... bugzilla--
16:30:49 melwitt oh cool, sorry for the double ping
16:31:01 lyarwood np at all
16:34:28 mriedem efried_rollin: want to drop the -W on https://review.openstack.org/#/c/649600/ ?
16:35:11 openstackgerrit Theodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.openstack.org/648686
16:37:37 openstackgerrit melanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556 https://review.openstack.org/562041
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Don't generate service UUID for deleted services https://review.openstack.org/582408
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1778305 https://review.openstack.org/582407
16:37:39 openstack bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)
16:37:40 openstack bug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Undecided,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)
16:46:00 mriedem mlavalle: did you sort out the NoValidHost error you were getting the other night?
16:46:38 mlavalle mriedem: yes, thanks for following up. it was the image filter
16:47:34 mriedem mlavalle: ok so you have some image with metadata that didn't match any hosts i'm guessing?
16:47:40 dansmith mriedem: were you going to un-WIP your stein backports?
16:47:45 mriedem dansmith: just finished
16:48:02 mlavalle mriedem: excatly and sisnce I have only one compute, that was the end of that request
16:48:05 dansmith okay gerrit didn't notify me
16:48:44 mriedem mlavalle: did enabling debug in the scheduler and checking the logs tell you what was up? or did you have to then look at the code too?
16:49:18 stephenfin mriedem: fwiw, your assumption that the issue I pointed out earlier wasn't ironic specific seems to have proven out. I spotted this after http://paste.openstack.org/show/fsWdgjpwsycPzdqPXNLI/
16:49:46 stephenfin Not sure why that's happening (I suspect something HA'y) but that would explain why ID, an auto-incremented primary key field (I assume) wasn't set
16:50:41 stephenfin I found a similar issue reported by MarkusZ years ago that I need to look into more, but that's tomorrow's work https://bugs.launchpad.net/nova/+bug/1566783/comments/4
16:50:43 openstack Launchpad bug 1566783 in OpenStack Compute (nova) "On service start, nova-compute fails to register itself to the DB" [Low,Confirmed]
16:50:45 mriedem stephenfin: that's a failure to create the compute node in the db because some other node with the same uuid already exists, but you wouldn't have a compute_nodes table record without an id
16:50:47 stephenfin Just closing that loop :)
16:50:59 mriedem so i still don't understand how you'd get that object action error on loading ComputeNode.id
16:51:10 mlavalle mriedem: without enabling debug, the conductor log gave the filter that was failing. at the end the conductor logs how many hosts each filter returns. All the filters looked something like: in: 1, out 1. The image filter was: in: 1, out: 0
16:51:45 stephenfin mriedem: I figured the initial object creation was failing so we're left with a Python object but no underlying DB entry
16:51:48 mriedem mlavalle: sure, just wondering if it was really hard to figure out but sounds like it wasn't
16:52:02 mriedem stephenfin: we shouldn't even get to _setup_pci_tracker in that case though
16:52:34 mlavalle mriedem: nop, just looking closer at the logs. I guess when I pinged you, it was Friday, night and I was tired..... Or maybe I just needed an excuse to say hi ;-)
16:53:10 mriedem stephenfin: i.e. if this fails https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L584 you're not going to be calling https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L589
16:53:34 mriedem the question would be why would we even try to create the compute node record if it already exists by uuid
16:53:59 mriedem because https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L567 should find the existing cn in the db
16:54:14 mriedem unless,
16:54:19 mriedem the hypervisor_hostname is different
16:54:47 stephenfin Could it be racy if you've multiple compute services running?
16:54:51 sean-k-mooney i was going to say we lookup the compute service by hostname to get the uuid
16:55:43 mriedem it could race if you've got multiple compute services trying to manage the same node yeah
16:56:19 sean-k-mooney or multiple nodes with the same hostname?
16:56:23 mriedem stephenfin: do you have this fix? https://github.com/openstack/nova/commit/57566f4c8d2929c25e76564883369a7c6eda720a
16:57:06 mriedem sean-k-mooney: i'm not sure what you're saying
16:57:16 mriedem "compute service by hostname to get the uuid" is wrong though in this case
16:57:36 mriedem compute nodes in the db are 1:1 with ironic nodes, and in this case we're looking up the compute node by nodename, i.e. hypervisor_hostname, i.e. ironic node.uuid
16:57:42 mriedem not the nova-compute service hostname
16:57:54 sean-k-mooney oh this is ironic never mind
16:58:03 stephenfin Damn it, I've to go (tag rugby match in 15 minutes). Sorry /o\
16:58:14 mriedem back to the pile
16:58:21 stephenfin yuuuup
16:59:12 sean-k-mooney i was going to say if there are two hosts with the same hostname in the cloud then the both endup sharing the same compute service and hypervior entry in the db and evertying is broken.
16:59:26 sean-k-mooney thats not related to that however

Earlier   Later