| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-15 | |||
| 18:12:36 | mriedem | imacdonn: in that case nova shouldn't try to delete the port | |
| 18:12:40 | mriedem | just unbind it | |
| 18:12:52 | gibi | dansmith: thanks for the reply on the request filter spec. I have a better view now, so I'm +1 | |
| 18:12:56 | mriedem | if nova didn't unbind it and a reschedule tried to bind that host to another host, that would fail | |
| 18:13:03 | dansmith | gibi: np, thanks for those comments | |
| 18:13:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Detach volumes when VM creation fails https://review.openstack.org/544144 | |
| 18:14:06 | imacdonn | mriedem: yeah. I'm fairly sure I've seen this with Ocata .. but it's one of those things that only happens when something else goes wrong (to cause the initial creation failure), so it never got hot enough for me to really dig into ... you just reminded me of if when you mentioned ports above | |
| 18:15:34 | mriedem | at the risk of angering the gods, i think we actually do a relatively decent job of managing ports and cleaning up after ourselves | |
| 18:15:39 | mriedem | with volumes, not so much | |
| 18:16:16 | imacdonn | heh. I'll try to find time to attempt to reproduce it ... but my "round tuit" supply is low | |
| 18:16:38 | mriedem | i understand | |
| 18:18:22 | imacdonn | I guess you really can by ANYTHING on Amazon! https://www.amazon.com/Round-TUIT-Tokens-Multi-Pack-Encouragement/dp/B00J8KVHLY | |
| 18:18:45 | mriedem | heh, find me a hang in there poster with a kitten that doesn't suck | |
| 18:18:58 | mriedem | and doesn't say "baby" on it | |
| 18:19:06 | imacdonn | added to my todo list .. I'm try when it .... well, you know... | |
| 18:19:14 | mriedem | https://play.google.com/store/books/details?id=KEMFmlv1uKcC&source=productsearch&utm_source=HA_Desktop_US&utm_medium=SEM&utm_campaign=PLA&pcampaignid=MKTAD0930BO1&gclid=Cj0KCQiA_JTUBRD4ARIsAL7_VeXOQT1TCYHJZb7ifQDAGqbTm2uSwLsNB0_bR7o0pTSXHwROSrbpsSUaAtxPEALw_wcB&gclsrc=aw.ds&dclid=CLC_ypvEqNkCFYzdwAod8RYOVg | |
| 18:19:58 | mriedem | oh there is one on amazon now, well goes to show demand is up since last i checked | |
| 18:20:43 | mriedem | jaypipes: even have one for you https://ih0.redbubble.net/image.475009417.6220/flat,800x800,070,f.u5.jpg | |
| 18:28:34 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add require_tenant_aggregate request filter https://review.openstack.org/545002 | |
| 18:28:35 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add a require_tenant_trait request filter https://review.openstack.org/545079 | |
| 18:28:55 | dansmith | efried: ^ example of using traits to do the tenant isolation as an alternative | |
| 18:29:06 | efried | dansmith: Nice, looking. | |
| 18:29:21 | dansmith | efried: and a question in there you can probably answer and save me a few minutes of reading | |
| 18:30:50 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Detach volumes when VM creation fails https://review.openstack.org/544143 | |
| 18:33:49 | melwitt | efried: ack, np | |
| 18:36:01 | melwitt | mriedem: yep, good point to avoid nova-net and cells v1 for mox -> mock | |
| 18:38:40 | efried | dansmith: Responded | |
| 18:40:16 | dansmith | efried: tanks | |
| 18:42:09 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Clean up volumes on boot failure https://review.openstack.org/545086 | |
| 18:42:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Detach volumes when VM creation fails https://review.openstack.org/545087 | |
| 18:42:44 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add require_tenant_trait request filter https://review.openstack.org/545079 | |
| 18:42:49 | TheJulia | dansmith: so we managed to get nova not to crash in ironic's grenade jobs.... we had to inject -B on to the python command line to prevent the .pyc files from being placed on disk | |
| 18:43:06 | dansmith | TheJulia: I heard.. that sounds a lot like a python bug to me | |
| 18:43:24 | dansmith | TheJulia: it's one thing if we get some broken call and an exception or something, but a segv seems way out of the realm of reasonable to me | |
| 18:43:46 | TheJulia | dansmith: I believe it is officially a feature.... | |
| 18:43:54 | dansmith | TheJulia: ... | |
| 18:44:42 | TheJulia | certian app toolsets allow dynamic recompliation/reloading of python code in the app during runtime, the trick afaik is to remove the .pyc file which was likely occuring during upgrade | |
| 18:44:49 | TheJulia | regardless, we're hunting something breaking with placement | |
| 18:45:14 | dansmith | TheJulia: right, but that can't cause a segv and not be called a bug, IMHO | |
| 18:45:29 | efried | dansmith: The request_spec.project_id is always a UUID, yes? | |
| 18:45:36 | dansmith | TheJulia: dynamic recompile is cool, even if it causes some python call imcompatibility or something, but not a segv | |
| 18:45:47 | dansmith | efried: I think it depends on your keystone backend, no? | |
| 18:45:58 | dansmith | TheJulia: what placement thing are you chasing now? | |
| 18:46:38 | dansmith | efried: I bet lbragstad knows | |
| 18:47:01 | dansmith | efried: https://github.com/midokura/python-midonetclient/issues/19 | |
| 18:47:22 | efried | dansmith: no idea, swhy I'm asking. Cause it's gonna make a difference how much you have to sanitize it, etc. | |
| 18:47:22 | efried | But | |
| 18:47:22 | efried | I think we may be barking up the wrong tree anyway. Don't we actually want the trait to be CUSTOM_HOST_AGGREGATE_{agg_id} ? | |
| 18:47:27 | lbragstad | efried it depends on the resource backend being used | |
| 18:47:58 | TheJulia | dansmith: http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-placement-api.txt.gz#_Feb_15_18_00_31_109054 after everything gets up and running and nova-compute is able to post data back out, we're getting a conflict, but I'm afraid we don't understand the mechanisms in that publishing/use of data | |
| 18:47:58 | dansmith | efried: I don't :) | |
| 18:48:14 | TheJulia | dansmith: to then go backwards and figure out what is truly causing that failure | |
| 18:48:47 | dansmith | TheJulia: hmm, that's interesting | |
| 18:48:50 | dansmith | jaypipes: ^ | |
| 18:49:13 | lbragstad | efried if keystone is told to pull projects from something other than the default sql backend, then we can't guarantee them to be uuids | |
| 18:49:23 | dansmith | TheJulia: I'm not sure that's really a blocking thing.. is there a matching failure in the n-cpu log? | |
| 18:49:32 | TheJulia | http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-n-cpu.txt.gz#_Feb_15_18_00_31_112744 | |
| 18:49:48 | efried | lbragstad: Thanks. | |
| 18:50:12 | lbragstad | efried yep | |
| 18:50:44 | dansmith | TheJulia: okay that almost maybe kinda looks like someone changed ids or there's some confusion going on | |
| 18:50:58 | dansmith | TheJulia: multiple nova-computes? multiple ironic nodes? | |
| 18:52:14 | dansmith | actually, efried ^ | |
| 18:52:15 | TheJulia | dansmith: 2x n-cpu running pike, 2x ironic-conductor (1 master, 1 queens) 1x ironic api running queens. | |
| 18:52:30 | jroll | and multiple ironic nodes | |
| 18:52:35 | TheJulia | yup | |
| 18:52:38 | TheJulia | 7 of them | |
| 18:52:40 | dansmith | efried: this is during a upgrade, any chance the report client being providertreeish after the upgrade is breaking something? | |
| 18:52:48 | dansmith | because it's ensuring that the provider exists, and then freaking out because it does | |
| 18:52:54 | jroll | reminder it could be a real incompatibility between pike nova and queens ironic, given this test has been off for weeks | |
| 18:52:58 | jroll | dansmith: still pike nova | |
| 18:53:08 | dansmith | jroll: ah, right | |
| 18:53:30 | TheJulia | jroll: it should presently be queens -> master, I'll double check (since the devstack-gate patch was involved) | |
| 18:53:47 | jroll | oh, right | |
| 18:53:53 | dansmith | okay so queens nova code yeah? | |
| 18:54:03 | jroll | which shouldn't be much different than queens/queens which has been working | |
| 18:54:06 | jroll | dansmith: yes, sorry | |
| 18:54:12 | jroll | queens nova on both sides | |
| 18:54:15 | dansmith | ack | |
| 18:54:38 | TheJulia | jroll: we should propably cherry-pick the patch to stable/queens and let it run to see what the case is there... | |
| 18:54:57 | jaypipes | mriedem: heh | |
| 18:54:58 | jroll | TheJulia: wouldn't hurt | |
| 18:55:11 | TheJulia | jroll: going to click button | |
| 18:55:15 | jroll | thank you | |
| 18:55:49 | dansmith | well, queens code still has the provider tree stuff so still worth efried looking at I think | |
| 18:56:06 | dansmith | although weird that it would be different/broken across such a small upgrade boundary | |
| 18:56:14 | jroll | yeah, that's my thought | |
| 18:56:15 | TheJulia | dansmith: in the mean time, we can check the pike -> queens job results and see if it is broken the same way | |
| 18:56:18 | efried | Yup, I'm looking. We shouldn't be able to get here except by really bad timing. | |
| 18:56:29 | efried | TheJulia: And you said it happened more than once? | |
| 18:56:39 | dansmith | more than once in the logs even | |
| 18:56:49 | dansmith | on each sync | |
| 18:56:58 | jroll | could also be a problem with our multinode, whether it's queens+queens or queens+master | |
| 18:57:00 | TheJulia | efried: we have _not_ tried to recheck in case it was some process fluke | |
| 18:57:22 | efried | No, this simply shouldn't happen, still looking... | |
| 19:00:12 | efried | Okay, this is a *name* conflict, not a *uuid* conflict. This means we somehow got two RPs with different UUIDs but the same name. | |
| 19:00:26 | efried | and the name is a UUID, which is nice and confusing. | |
| 19:00:35 | cfriesen | mriedem: with respect to https://review.openstack.org/#/c/544748/ and my proposed change https://review.openstack.org/#/c/525253/. In our case it passes scheduling but then fails for whatever reason on the compute node. As such, the proposed fix is not sufficient because we would not end up calling _bury_in_cell0(). | |
| 19:00:36 | efried | I thought johnthetubaguy was banging his head against this last Fall. | |
| 19:00:46 | jroll | efried: the name would be the ironic uuid, right? | |
| 19:01:01 | efried | jroll: I'm not an expert there, but yeah, something like that. | |