Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-10
14:36:55 mriedem just making sure nova+neutron+placement are all doing what they are supposed to (from the gate) would be a good step anyway
14:37:11 gibi mriedem: yeah I think we have such tempest test on review
14:37:15 gibi let me look up the link
14:37:38 gibi mriedem: https://review.openstack.org/#/c/629253/
14:38:29 gibi for the heal port allocation we would need to create a bound port in rocky that would need allocation in stein
14:38:51 gibi unfortunately in rocky only sriov ports supported min bw in neutron, and sriov is no no in the gate
14:39:23 gibi I can trick the system by adding the min bw policy to the port _after_ it was bound but that feels hackis already
14:39:35 mriedem agree
14:39:46 gibi I think I wil stick to the functional test env for heal port allocation
14:40:09 gibi and I might do some manual test in a devstack where I have sriov
14:46:54 mriedem melwitt: are you ok to re-approve the cross-cell resize spec https://review.openstack.org/#/c/642807/ ? I saw you were just +1. I'd like to get the blueprint re-approved so i can start putting the bottom tail of the (huge) series into runways to get some reviews going.
14:53:45 stephenfin mriedem: Fancy revisiting this? https://review.openstack.org/#/c/650018/
14:55:39 mriedem "I've spoken with Lars Kurth (XenProject advisory board chair) and he agrees that at this point in time, this config can be removed."
14:55:46 mriedem so we have no CI for libvirt+xen
14:55:49 mriedem and no plans to
14:55:50 mriedem ok
14:57:53 mriedem stephenfin: back at you https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1823781
14:58:07 stephenfin ack
14:59:17 stephenfin mriedem: Out of curiosity, does this look familiar to you? http://paste.openstack.org/show/P6AlRRpBWnYwFfWWVJFl/ It's a stable/rocky deployment
15:00:44 mriedem yes i think so, sec
15:01:29 mriedem actually no nvm, was thinking of something else pci related
15:02:32 mriedem it's blowing up on this? "n_id = compute_node.id"
15:02:43 mriedem i only see _setup_pci_tracker called with an existing compute node record where id should be set
15:02:47 mriedem unless you guys have a fork
15:04:53 stephenfin Not that I'm aware of. I had been doing some work in that area but none of that is merged yet :/ https://review.openstack.org/#/c/649559/
15:05:54 stephenfin But yeah, looks like that. I don't know if that attribute should be set or not. I can't see why it wouldn't but all things Ironic really aren't my strong suit
15:06:20 melwitt mriedem: I was thinking dansmith would want to sign off on it
15:08:58 mriedem stephenfin: i don't think that error has anything to do with ironic
15:13:21 stephenfin Perhaps, but it's only happening with baremetal nodes so I suspect it's some with how we're initializing the Ironic driver
15:13:25 stephenfin but that's a grasp
15:15:01 mriedem so the driver returns a list of nodenames for that host and those get passed one at a time to RT https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7893
15:16:50 mriedem should eventually call through to here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L743
15:17:24 mriedem if the compute node already exists in the db you'd get here on first startup https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L567
15:18:08 mriedem this should also be ok https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L530
15:18:35 mriedem otherwise we'd create the node in the db https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L584
15:18:38 mriedem which should set the id on the cn object
15:19:07 mriedem https://github.com/openstack/nova/blob/stable/rocky/nova/objects/compute_node.py#L322
15:19:50 mriedem looks like you've got some debugging to do if you can recreate it
15:32:12 mriedem lyarwood: melwitt: dansmith: bauzas: looks like we can start approving stable/stein backports now so i'll start going through those and removing the WIP on mine
15:32:27 mriedem we need to roll through some of those to get things flushed out down through pike for pike-em
15:32:38 dansmith ack
15:32:40 melwitt ah, yep
15:33:28 melwitt I have a bug fix that needs to go back to pike but it hasn't even merged on master yet and having trouble getting review https://review.openstack.org/611974
15:34:19 cdent mriedem: does a server always get buried to cell0 when it fails to find a host? Is it possible to say "only bury if you requested a build at a microversion that will allow an unbury"?
15:34:30 mriedem melwitt: i definitely don't know that code so would need at least lyarwood and/or mdbooth to sign off
15:34:43 mriedem cdent: "does a server always get buried to cell0 when it fails to find a host?" yes
15:35:05 mriedem "Is it possible to say "only bury if you requested a build at a microversion that will allow an unbury"?" i think that gets really complicated fast
15:35:12 cdent i imagine so
15:35:20 cdent what was the original purpose of burying?
15:35:22 dansmith yeah that's not okay
15:35:27 melwitt there's this one too but people were meh about reviewing it too https://review.openstack.org/582408
15:35:27 mriedem because it's more like (1) only bury if using a microversion to allow unburying except (2) unless you didn't request personality files etc
15:36:00 mriedem cdent: need to be able to show the servers from the API even though they are in error state and not on a host,
15:36:01 dansmith cdent: it's still the original purpose, which is that if we didn't pick a host, we didn't pick a cell, thus we have nowhere to store it
15:36:12 mriedem and the api pulls servers from a cell db
15:36:39 mriedem we have build_requests but those are short-lived
15:36:44 cdent ah, right, I keep forgetting that a server that is an ERROR is still a server (which is a decision I've never quite understood)
15:37:14 mriedem melwitt: it's failing pep8 :)
15:37:38 bauzas mriedem: ack, sorry on a meeting
15:38:43 melwitt sigh
15:40:52 openstackgerrit Surya Seetharaman proposed openstack/nova master: Block automatic transport_url update for cell0 https://review.openstack.org/605414
15:46:38 openstackgerrit Merged openstack/nova-specs master: Re-propose cross-cell-resize spec for Train https://review.openstack.org/642807
16:22:27 bauzas melwitt: thanks for reviewing https://review.openstack.org/#/c/650963/ I replied to your question
16:23:24 melwitt bauzas: a-ha, thanks
16:30:00 melwitt lyarwood, mdbooth: begging for reviews on this again, disk unit stuff https://review.openstack.org/611974
16:30:37 lyarwood melwitt: ack saw above, I'll take a look once I've finished with all of this downstream paperwork... bugzilla--
16:30:49 melwitt oh cool, sorry for the double ping
16:31:01 lyarwood np at all
16:34:28 mriedem efried_rollin: want to drop the -W on https://review.openstack.org/#/c/649600/ ?
16:35:11 openstackgerrit Theodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.openstack.org/648686
16:37:37 openstackgerrit melanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556 https://review.openstack.org/562041
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Don't generate service UUID for deleted services https://review.openstack.org/582408
16:37:39 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1778305 https://review.openstack.org/582407
16:37:39 openstack bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)
16:37:40 openstack bug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Undecided,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)
16:46:00 mriedem mlavalle: did you sort out the NoValidHost error you were getting the other night?
16:46:38 mlavalle mriedem: yes, thanks for following up. it was the image filter
16:47:34 mriedem mlavalle: ok so you have some image with metadata that didn't match any hosts i'm guessing?
16:47:40 dansmith mriedem: were you going to un-WIP your stein backports?
16:47:45 mriedem dansmith: just finished
16:48:02 mlavalle mriedem: excatly and sisnce I have only one compute, that was the end of that request
16:48:05 dansmith okay gerrit didn't notify me
16:48:44 mriedem mlavalle: did enabling debug in the scheduler and checking the logs tell you what was up? or did you have to then look at the code too?
16:49:18 stephenfin mriedem: fwiw, your assumption that the issue I pointed out earlier wasn't ironic specific seems to have proven out. I spotted this after http://paste.openstack.org/show/fsWdgjpwsycPzdqPXNLI/
16:49:46 stephenfin Not sure why that's happening (I suspect something HA'y) but that would explain why ID, an auto-incremented primary key field (I assume) wasn't set
16:50:41 stephenfin I found a similar issue reported by MarkusZ years ago that I need to look into more, but that's tomorrow's work https://bugs.launchpad.net/nova/+bug/1566783/comments/4
16:50:43 openstack Launchpad bug 1566783 in OpenStack Compute (nova) "On service start, nova-compute fails to register itself to the DB" [Low,Confirmed]
16:50:45 mriedem stephenfin: that's a failure to create the compute node in the db because some other node with the same uuid already exists, but you wouldn't have a compute_nodes table record without an id
16:50:47 stephenfin Just closing that loop :)
16:50:59 mriedem so i still don't understand how you'd get that object action error on loading ComputeNode.id
16:51:10 mlavalle mriedem: without enabling debug, the conductor log gave the filter that was failing. at the end the conductor logs how many hosts each filter returns. All the filters looked something like: in: 1, out 1. The image filter was: in: 1, out: 0
16:51:45 stephenfin mriedem: I figured the initial object creation was failing so we're left with a Python object but no underlying DB entry
16:51:48 mriedem mlavalle: sure, just wondering if it was really hard to figure out but sounds like it wasn't
16:52:02 mriedem stephenfin: we shouldn't even get to _setup_pci_tracker in that case though
16:52:34 mlavalle mriedem: nop, just looking closer at the logs. I guess when I pinged you, it was Friday, night and I was tired..... Or maybe I just needed an excuse to say hi ;-)
16:53:10 mriedem stephenfin: i.e. if this fails https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L584 you're not going to be calling https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L589
16:53:34 mriedem the question would be why would we even try to create the compute node record if it already exists by uuid
16:53:59 mriedem because https://github.com/openstack/nova/blob/stable/rocky/nova/compute/resource_tracker.py#L567 should find the existing cn in the db
16:54:14 mriedem unless,
16:54:19 mriedem the hypervisor_hostname is different

Earlier   Later