| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-31 | |||
| 17:12:36 | openstackgerrit | Lee Yarwood proposed openstack/nova: libvirt: Limit destroying disks during cleanup to spawn https://review.openstack.org/427267 | |
| 17:14:07 | openstackgerrit | Chris Dent proposed openstack/nova: WIP: Explore functional test for the resource tracker https://review.openstack.org/427309 | |
| 17:15:04 | mgould | hi everyone | |
| 17:15:06 | mgould | I'm getting "Host is not mapped to any cell" errors in nova boot on a clean devstack on a freshly-installed baremetal machine | |
| 17:15:28 | mgould | I've previously used the same local.conf on a VM without issues | |
| 17:15:34 | mgould | any idea what I've done wrong? | |
| 17:15:34 | openstack | bug 1646896 in OpenStack Compute (nova) "System hangs when using NFS storage backend with loopback mounts" [Low,Confirmed] https://launchpad.net/bugs/1646896 | |
| 17:15:34 | openstackgerrit | Mihail proposed openstack/nova: proposed fix for bug 1646896 - System hangs when using NFS storage backend with loopback mounts, Closes-Bug: #1646896 https://review.openstack.org/422041 | |
| 17:16:07 | cdent | jaypipes, edleafe, johnthetubaguy, mriedem : that review ^ about functional test for resource provider is trying to the basis for one that tests ironic. at the moment is just does non ironic | |
| 17:16:14 | cdent | I needed to start there to get any traction | |
| 17:17:00 | jaypipes | cdent: hmm, I was right in the middle of adding a functional test to the Ironic resource classes patch. | |
| 17:17:18 | cdent | ah, good, now we'll have more | |
| 17:17:21 | cdent | ;) | |
| 17:17:42 | cdent | that one is very bland | |
| 17:18:56 | cdent | jaypipes: perhaps we can compare and contrast later. I'm going to fix a couple bugs I found while making that. | |
| 17:22:24 | jaypipes | cdent: k | |
| 17:22:50 | cdent | if my version is for naught, was an excellent learning exercise | |
| 17:27:26 | openstackgerrit | Diana Clarke proposed openstack/nova: Fix map_cell_and_hosts help https://review.openstack.org/422171 | |
| 17:28:00 | openstackgerrit | Chris Dent proposed openstack/nova: Fresh resource provider in RT must have generation 0 https://review.openstack.org/427319 | |
| 17:40:38 | openstackgerrit | Chris Dent proposed openstack/nova: [placement] Use modern attributes of oslo_context https://review.openstack.org/427330 | |
| 17:41:36 | cdent | mriedem: both of those ^ might be rc oriented | |
| 17:41:57 | mriedem | i +2ed the first one | |
| 17:42:18 | mriedem | https://review.openstack.org/#/c/427319/ isn't a regression in ocata though right? it was just always that way | |
| 17:47:48 | cdent | mriedem: not a regression, no, but sending out something with noise on it seems untidy | |
| 17:50:09 | mriedem | sure, but we're also 2 days from rc1 | |
| 17:50:17 | mriedem | so i can live with untidy | |
| 17:50:21 | mriedem | since we can backport fixes | |
| 17:50:32 | mriedem | anyway, i'm +2 on it since it's a simple fix | |
| 17:50:54 | cdent | mriedem: that's cool. All this release schedule stuff is voodoo to me. I just smile and nod. | |
| 17:50:57 | dansmith | jroll: sdague: so, we really need to be running discover_hosts after compute nodes are in the database, which for ironic happens pretty late compared to other virt drivers | |
| 17:51:17 | dansmith | jroll: sdague: should I try to wedge in a discover_hosts call into the late stages of ironic devstack stuff? | |
| 17:52:28 | dansmith | jroll: sdague: or maybe it's futile if people don't provide ironic nodes at the time of a devstack run? I'm not actually sure how that works | |
| 17:55:52 | mriedem | cdent: it's pretty simple, just don't risk putting anything in right before rc1 unless it's super low risk, or super critical to get in | |
| 17:56:07 | mriedem | since you don't want to release a major version with some kind of major bug which prevents upgrades or something | |
| 17:56:09 | mriedem | ha | |
| 17:56:30 | cdent | I understand the mechanics. | |
| 17:56:36 | mriedem | having said all that, i'd bet most deployments wait much longer to pick up anything, | |
| 17:56:42 | cdent | that | |
| 17:56:44 | mriedem | so when they upgrade, they're going to pick up like the 3rd stable branch release anyway | |
| 18:01:11 | mriedem | cdent: before you dine, | |
| 18:01:18 | cdent | mriedem: yessir? | |
| 18:01:28 | mriedem | did either you or bauzas put anything up for the placement upgrade stuff in the devref? if not, i can start that | |
| 18:01:31 | mriedem | it will be pretty simple | |
| 18:01:34 | mriedem | just upgrade from newton notes | |
| 18:01:40 | cdent | I did not | |
| 18:01:43 | mriedem | bauzas: ? | |
| 18:01:58 | mriedem | ok i'll get started on that one | |
| 18:02:04 | cdent | thanks | |
| 18:10:19 | dansmith | sdague: question for you here: https://review.openstack.org/#/c/426310 | |
| 18:12:07 | sdague | dansmith: ok, question away | |
| 18:12:17 | dansmith | sdague: it's in the review | |
| 18:12:26 | dansmith | about verify_noapi | |
| 18:14:26 | sdague | how can this run when the api is down? | |
| 18:14:35 | dansmith | sdague: there are other ways to solve this, including some sort of post phase for the plugin, where we've already run the verify, if you'd prefer | |
| 18:14:38 | dansmith | sdague: it's nova-manage | |
| 18:14:40 | sdague | it all depends on 'nova service-list' | |
| 18:14:59 | dansmith | oh, did I do it with service-list? I did.. | |
| 18:15:33 | dansmith | I don't actually care that the api is down, of course, although I guess that means we won't have the primary node compute up either | |
| 18:15:45 | sdague | right, that too | |
| 18:15:53 | dansmith | so, then, yeah, I need to run discover_hosts again after we've done the resource verify | |
| 18:17:04 | sdague | so, given the number of other things in flight, does this catch a failure that would have passed otherwise? Or is this just catching a failure earlier? | |
| 18:17:07 | dansmith | sdague: so, according to the plugin docs, I could run discover from resource verify post-upgrade I guess | |
| 18:17:36 | sdague | dansmith: yeh, post-upgrade seems like a put to run the discover_hosts again if you need it | |
| 18:17:38 | dansmith | sdague: this is a new thing we've introduced, and that log link proves that we're actually hitting it | |
| 18:18:12 | sdague | ok, but the fix for that isn't this, right? | |
| 18:18:22 | sdague | the fix for that is another discover_hosts call | |
| 18:18:37 | dansmith | sdague: no, this is the fix, otherwise the new discover_hosts call is still a race | |
| 18:18:50 | dansmith | sdague: if both computes aren't up, the discover_hosts won't do anything | |
| 18:19:52 | dansmith | sdague: am I missing why you don't like this service check? | |
| 18:23:21 | sdague | dansmith: so where is the discover_hosts getting run? | |
| 18:23:45 | dansmith | sdague: right now it's only run in simple_cell_setup | |
| 18:24:03 | openstackgerrit | OpenStack Release Bot proposed openstack/os-vif: Update reno for stable/ocata https://review.openstack.org/427341 | |
| 18:24:56 | sdague | dansmith: and where does that get called during this flow that is failing? | |
| 18:25:20 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif: Updated from global requirements https://review.openstack.org/395347 | |
| 18:25:52 | dansmith | sdague: I'm not sure what you mean.. it's not failing, it's just not seeing the computenode record for the subnode when it runs in simple_cell_setup, which means later we end up without the hostmapping, and thus the failed test in that log I linked | |
| 18:25:54 | openstackgerrit | OpenStack Release Bot proposed openstack/python-novaclient: Update reno for stable/ocata https://review.openstack.org/427344 | |
| 18:26:33 | sdague | dansmith: right... but the issue is that's a failure here - https://github.com/openstack-dev/grenade/blob/a1ea5cec068d59b9c2caa1c82d7ee2584d63aa08/grenade.sh#L270 | |
| 18:26:51 | sdague | but all this code is executed here - https://github.com/openstack-dev/grenade/blob/a1ea5cec068d59b9c2caa1c82d7ee2584d63aa08/grenade.sh#L283-L287 | |
| 18:27:14 | sdague | and I don't understand how code that runs after the failure in question is going to fix the issue | |
| 18:27:41 | dansmith | sdague: you mean it's on the old side? | |
| 18:27:47 | sdague | yes | |
| 18:28:08 | sdague | http://logs.openstack.org/79/396679/45/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/11607f2/logs/grenade.sh.txt.gz#_2017-01-30_09_49_58_107 | |
| 18:28:16 | dansmith | sdague: okay, I'm not sure how it's hitting that on the old side, since the thing that is failing merged on friday | |
| 18:28:26 | sdague | dansmith: well that's where this failure is | |
| 18:28:51 | dansmith | sdague: okay, so, I guess we'll continue to hold onto this fix for a condition that we know we've added | |
| 18:29:07 | dansmith | sdague: if you could tell me why you have a problem with it, it'd sure save us time when this becomes a critical problem | |
| 18:29:40 | dansmith | 'cause I still am confused about that part | |
| 18:30:15 | sdague | dansmith: because I don't see what it's making better. This was framed with the logs in question as fixing that issue, but they don't, because that's a different and real issue about initial node racing on the way up | |
| 18:30:57 | artom | Is http://logs.openstack.org/59/424759/12/gate/gate-tempest-dsvm-cells-ubuntu-xenial/d7b1311/console.html#_2017-01-31_17_48_34_663215 something to do with ongoing cells work? | |
| 18:30:58 | dansmith | sdague: you agree that race conditions exist that we know about, but just haven't hit yet right? | |
| 18:32:59 | sdague | dansmith: there are bunches of those though, and I'm fine fixing issues we're hitting. It just seems like this keeps getting pushed for issues we aren't yet hitting. And that are unlikely to hit based on the real timelines for additional stuff that happens before the second tempest run happens | |
| 18:33:58 | dansmith | sdague: right, so paraphrasing, what you just said is: "This is a race we know about, but right now we think we'll never hit it, so we'll just wait until the context has fallen out of our brains and it becomes a gate blocker in six months" | |
| 18:34:04 | dansmith | over ten lines of bash | |
| 18:34:36 | dansmith | sdague: and fwiw, the reason they're hitting that (just found out) is that their patch set base to master | |
| 18:34:51 | dansmith | hitting that in that way I mean | |
| 18:42:42 | melwitt | artom: not that I know of | |
| 18:43:38 | artom | melwitt, hrmm, thanks :) | |
| 18:47:35 | melwitt | artom: we recently merged a patch (on Jan 27) that might be related to this, so I'll do some digging through the logs to see what could be happening | |
| 18:48:38 | artom | melwitt, 4 days ago? We'd have noticed it by now if it was causing consistent failure in the gate... | |
| 18:48:48 | mriedem | i've seen that failure | |