| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-15 | |||
| 18:20:43 | mriedem | jaypipes: even have one for you https://ih0.redbubble.net/image.475009417.6220/flat,800x800,070,f.u5.jpg | |
| 18:28:34 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add require_tenant_aggregate request filter https://review.openstack.org/545002 | |
| 18:28:35 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add a require_tenant_trait request filter https://review.openstack.org/545079 | |
| 18:28:55 | dansmith | efried: ^ example of using traits to do the tenant isolation as an alternative | |
| 18:29:06 | efried | dansmith: Nice, looking. | |
| 18:29:21 | dansmith | efried: and a question in there you can probably answer and save me a few minutes of reading | |
| 18:30:50 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Detach volumes when VM creation fails https://review.openstack.org/544143 | |
| 18:33:49 | melwitt | efried: ack, np | |
| 18:36:01 | melwitt | mriedem: yep, good point to avoid nova-net and cells v1 for mox -> mock | |
| 18:38:40 | efried | dansmith: Responded | |
| 18:40:16 | dansmith | efried: tanks | |
| 18:42:09 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Clean up volumes on boot failure https://review.openstack.org/545086 | |
| 18:42:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Detach volumes when VM creation fails https://review.openstack.org/545087 | |
| 18:42:44 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add require_tenant_trait request filter https://review.openstack.org/545079 | |
| 18:42:49 | TheJulia | dansmith: so we managed to get nova not to crash in ironic's grenade jobs.... we had to inject -B on to the python command line to prevent the .pyc files from being placed on disk | |
| 18:43:06 | dansmith | TheJulia: I heard.. that sounds a lot like a python bug to me | |
| 18:43:24 | dansmith | TheJulia: it's one thing if we get some broken call and an exception or something, but a segv seems way out of the realm of reasonable to me | |
| 18:43:46 | TheJulia | dansmith: I believe it is officially a feature.... | |
| 18:43:54 | dansmith | TheJulia: ... | |
| 18:44:42 | TheJulia | certian app toolsets allow dynamic recompliation/reloading of python code in the app during runtime, the trick afaik is to remove the .pyc file which was likely occuring during upgrade | |
| 18:44:49 | TheJulia | regardless, we're hunting something breaking with placement | |
| 18:45:14 | dansmith | TheJulia: right, but that can't cause a segv and not be called a bug, IMHO | |
| 18:45:29 | efried | dansmith: The request_spec.project_id is always a UUID, yes? | |
| 18:45:36 | dansmith | TheJulia: dynamic recompile is cool, even if it causes some python call imcompatibility or something, but not a segv | |
| 18:45:47 | dansmith | efried: I think it depends on your keystone backend, no? | |
| 18:45:58 | dansmith | TheJulia: what placement thing are you chasing now? | |
| 18:46:38 | dansmith | efried: I bet lbragstad knows | |
| 18:47:01 | dansmith | efried: https://github.com/midokura/python-midonetclient/issues/19 | |
| 18:47:22 | efried | dansmith: no idea, swhy I'm asking. Cause it's gonna make a difference how much you have to sanitize it, etc. | |
| 18:47:22 | efried | But | |
| 18:47:22 | efried | I think we may be barking up the wrong tree anyway. Don't we actually want the trait to be CUSTOM_HOST_AGGREGATE_{agg_id} ? | |
| 18:47:27 | lbragstad | efried it depends on the resource backend being used | |
| 18:47:58 | TheJulia | dansmith: http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-placement-api.txt.gz#_Feb_15_18_00_31_109054 after everything gets up and running and nova-compute is able to post data back out, we're getting a conflict, but I'm afraid we don't understand the mechanisms in that publishing/use of data | |
| 18:47:58 | dansmith | efried: I don't :) | |
| 18:48:14 | TheJulia | dansmith: to then go backwards and figure out what is truly causing that failure | |
| 18:48:47 | dansmith | TheJulia: hmm, that's interesting | |
| 18:48:50 | dansmith | jaypipes: ^ | |
| 18:49:13 | lbragstad | efried if keystone is told to pull projects from something other than the default sql backend, then we can't guarantee them to be uuids | |
| 18:49:23 | dansmith | TheJulia: I'm not sure that's really a blocking thing.. is there a matching failure in the n-cpu log? | |
| 18:49:32 | TheJulia | http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-n-cpu.txt.gz#_Feb_15_18_00_31_112744 | |
| 18:49:48 | efried | lbragstad: Thanks. | |
| 18:50:12 | lbragstad | efried yep | |
| 18:50:44 | dansmith | TheJulia: okay that almost maybe kinda looks like someone changed ids or there's some confusion going on | |
| 18:50:58 | dansmith | TheJulia: multiple nova-computes? multiple ironic nodes? | |
| 18:52:14 | dansmith | actually, efried ^ | |
| 18:52:15 | TheJulia | dansmith: 2x n-cpu running pike, 2x ironic-conductor (1 master, 1 queens) 1x ironic api running queens. | |
| 18:52:30 | jroll | and multiple ironic nodes | |
| 18:52:35 | TheJulia | yup | |
| 18:52:38 | TheJulia | 7 of them | |
| 18:52:40 | dansmith | efried: this is during a upgrade, any chance the report client being providertreeish after the upgrade is breaking something? | |
| 18:52:48 | dansmith | because it's ensuring that the provider exists, and then freaking out because it does | |
| 18:52:54 | jroll | reminder it could be a real incompatibility between pike nova and queens ironic, given this test has been off for weeks | |
| 18:52:58 | jroll | dansmith: still pike nova | |
| 18:53:08 | dansmith | jroll: ah, right | |
| 18:53:30 | TheJulia | jroll: it should presently be queens -> master, I'll double check (since the devstack-gate patch was involved) | |
| 18:53:47 | jroll | oh, right | |
| 18:53:53 | dansmith | okay so queens nova code yeah? | |
| 18:54:03 | jroll | which shouldn't be much different than queens/queens which has been working | |
| 18:54:06 | jroll | dansmith: yes, sorry | |
| 18:54:12 | jroll | queens nova on both sides | |
| 18:54:15 | dansmith | ack | |
| 18:54:38 | TheJulia | jroll: we should propably cherry-pick the patch to stable/queens and let it run to see what the case is there... | |
| 18:54:57 | jaypipes | mriedem: heh | |
| 18:54:58 | jroll | TheJulia: wouldn't hurt | |
| 18:55:11 | TheJulia | jroll: going to click button | |
| 18:55:15 | jroll | thank you | |
| 18:55:49 | dansmith | well, queens code still has the provider tree stuff so still worth efried looking at I think | |
| 18:56:06 | dansmith | although weird that it would be different/broken across such a small upgrade boundary | |
| 18:56:14 | jroll | yeah, that's my thought | |
| 18:56:15 | TheJulia | dansmith: in the mean time, we can check the pike -> queens job results and see if it is broken the same way | |
| 18:56:18 | efried | Yup, I'm looking. We shouldn't be able to get here except by really bad timing. | |
| 18:56:29 | efried | TheJulia: And you said it happened more than once? | |
| 18:56:39 | dansmith | more than once in the logs even | |
| 18:56:49 | dansmith | on each sync | |
| 18:56:58 | jroll | could also be a problem with our multinode, whether it's queens+queens or queens+master | |
| 18:57:00 | TheJulia | efried: we have _not_ tried to recheck in case it was some process fluke | |
| 18:57:22 | efried | No, this simply shouldn't happen, still looking... | |
| 19:00:12 | efried | Okay, this is a *name* conflict, not a *uuid* conflict. This means we somehow got two RPs with different UUIDs but the same name. | |
| 19:00:26 | efried | and the name is a UUID, which is nice and confusing. | |
| 19:00:35 | cfriesen | mriedem: with respect to https://review.openstack.org/#/c/544748/ and my proposed change https://review.openstack.org/#/c/525253/. In our case it passes scheduling but then fails for whatever reason on the compute node. As such, the proposed fix is not sufficient because we would not end up calling _bury_in_cell0(). | |
| 19:00:36 | efried | I thought johnthetubaguy was banging his head against this last Fall. | |
| 19:00:46 | jroll | efried: the name would be the ironic uuid, right? | |
| 19:01:01 | efried | jroll: I'm not an expert there, but yeah, something like that. | |
| 19:01:08 | dansmith | efried: I wonder if we detected the ironic node go away and come back and we're trying to recreate it but never deleted it? | |
| 19:01:14 | dansmith | jroll: yes | |
| 19:01:32 | jroll | I seem to remember this being john's head banging https://review.openstack.org/#/c/508555/ | |
| 19:02:11 | jroll | but there shouldn't be a rebalance happening | |
| 19:02:46 | efried | jroll: Beat me to it, yeah, that's the patch I was thinking of. | |
| 19:03:01 | efried | but there was more to it than that, I thought. | |
| 19:03:32 | jroll | not sure | |
| 19:04:15 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: mirror nova host aggregates to placement API https://review.openstack.org/545057 | |
| 19:04:22 | jroll | the title on the bug certainly looks like this problem: https://bugs.launchpad.net/nova/+bug/1714248 | |
| 19:04:24 | openstack | Launchpad bug 1714248 in OpenStack Compute (nova) pike "Compute node HA for ironic doesn't work due to the name duplication of Resource Provider " [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 19:04:33 | efried | There was another patch where we mucked with one of the rt update methods to look for things with different names. | |
| 19:04:47 | dansmith | jroll: if ironic was down and the two nova computes notice it is back at different times maybe they could be disagreeing briefly on who owns what? | |
| 19:05:11 | jroll | dansmith: yeah, trying to track down if it's long enough to trigger a rebalance | |
| 19:05:58 | jroll | servicegroup api tells us | |
| 19:06:02 | melwitt | mriedem: FYI, digging into the difference between the cleanup volumes vs ports bugs today | |
| 19:06:18 | melwitt | based on your comments in the patch | |
| 19:08:50 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: DNM: Test LM with encrypted volumes https://review.openstack.org/545093 | |