| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-15 | |||
| 17:59:09 | mriedem | deleting the volume before we reschedule is kind of dumb | |
| 17:59:24 | mriedem | let's say i boot from volume with a pre-existing volume, | |
| 17:59:32 | gibi | mriedem: oops, +Wd it | |
| 17:59:36 | mriedem | spawn on the first host fails after i've attached the instance to the volume, | |
| 17:59:47 | mriedem | we then detach and delete the volume, and reschedule, | |
| 17:59:55 | mriedem | then we get to the 2nd host, and attach will fail b/c the volume is gone | |
| 18:00:05 | mriedem | well i was +2, | |
| 18:00:08 | mriedem | and might still be, | |
| 18:00:27 | mriedem | because this is a latent thing that's always been in here, but i just don't think it's ever worked because of the bug he's fixing | |
| 18:00:51 | gibi | mriedem: I see. I guess this is a new bug report then | |
| 18:02:40 | gibi | mriedem: we fixed something that allowed another latent bug to manifest itself | |
| 18:03:02 | mriedem | at least delete_on_termination defaults to False... | |
| 18:03:15 | mriedem | this is also kind of why i didn't like the spec proposed for doing delete_on_termination with ports | |
| 18:03:22 | mriedem | since nova is not good at orchestrating this | |
| 18:04:28 | gibi | I also think that nova should only delete these if nova was the one that created them in the first place. But it would be even better if nova would never create these things | |
| 18:04:43 | mriedem | yeah that's the exact thing i was thinking | |
| 18:04:51 | mriedem | and we handle that with ports by storing a flag if nova created the port or not | |
| 18:05:00 | mriedem | in the nw info cache | |
| 18:05:33 | mriedem | because if nova did create this volume and attach it, then we fail and reschedule but don't delete it, we'll create another volume on the next host we try | |
| 18:05:38 | mriedem | and never cleanup the old one | |
| 18:06:04 | mriedem | and given enough retries, you could also hit quota limits on creating volumes | |
| 18:06:32 | mriedem | oh, well we do know if nova created it actually, | |
| 18:06:38 | mriedem | if source_type=volume, it's pre-created | |
| 18:08:26 | mriedem | ok i left some notes in the review so we can remember this conversation 2 years from now | |
| 18:08:30 | mriedem | when it's still a bug :) | |
| 18:09:23 | imacdonn | was something fixed recently-ish w.r.t. ports on reschedule .... seems I've seen issues with an exception, something like PortInUse, when an instance creation fails and gets rescheduled on another compute node, but that probably was with Ocata | |
| 18:09:25 | openstackgerrit | Merged openstack/nova-specs master: Add placement-req-filter spec https://review.openstack.org/544585 | |
| 18:10:42 | mriedem | imacdonn: that's pretty vague | |
| 18:10:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM: Test LM with encrypted volumes https://review.openstack.org/536350 | |
| 18:10:47 | mriedem | https://github.com/openstack/nova/commit/08d24b733ee9f4da44bfbb8d6d3914924a41ccdc went into newton | |
| 18:10:52 | mriedem | *mitaka | |
| 18:10:52 | gibi | mriedem: thanks for that note | |
| 18:11:15 | mriedem | imacdonn: i have this https://review.openstack.org/#/c/520248/ | |
| 18:11:21 | mriedem | but might not be what you're asking about | |
| 18:11:29 | imacdonn | mriedem: yeah, I know ... I'd have to try to come up with a way to reproduce it ... but it's likely that the port was not created by nova, in my case .. because I use heat a lot, and usually create the port separately there | |
| 18:11:31 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: DNM: Test LM with encrypted volumes https://review.openstack.org/545074 | |
| 18:12:36 | mriedem | imacdonn: in that case nova shouldn't try to delete the port | |
| 18:12:40 | mriedem | just unbind it | |
| 18:12:52 | gibi | dansmith: thanks for the reply on the request filter spec. I have a better view now, so I'm +1 | |
| 18:12:56 | mriedem | if nova didn't unbind it and a reschedule tried to bind that host to another host, that would fail | |
| 18:13:03 | dansmith | gibi: np, thanks for those comments | |
| 18:13:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Detach volumes when VM creation fails https://review.openstack.org/544144 | |
| 18:14:06 | imacdonn | mriedem: yeah. I'm fairly sure I've seen this with Ocata .. but it's one of those things that only happens when something else goes wrong (to cause the initial creation failure), so it never got hot enough for me to really dig into ... you just reminded me of if when you mentioned ports above | |
| 18:15:34 | mriedem | at the risk of angering the gods, i think we actually do a relatively decent job of managing ports and cleaning up after ourselves | |
| 18:15:39 | mriedem | with volumes, not so much | |
| 18:16:16 | imacdonn | heh. I'll try to find time to attempt to reproduce it ... but my "round tuit" supply is low | |
| 18:16:38 | mriedem | i understand | |
| 18:18:22 | imacdonn | I guess you really can by ANYTHING on Amazon! https://www.amazon.com/Round-TUIT-Tokens-Multi-Pack-Encouragement/dp/B00J8KVHLY | |
| 18:18:45 | mriedem | heh, find me a hang in there poster with a kitten that doesn't suck | |
| 18:18:58 | mriedem | and doesn't say "baby" on it | |
| 18:19:06 | imacdonn | added to my todo list .. I'm try when it .... well, you know... | |
| 18:19:14 | mriedem | https://play.google.com/store/books/details?id=KEMFmlv1uKcC&source=productsearch&utm_source=HA_Desktop_US&utm_medium=SEM&utm_campaign=PLA&pcampaignid=MKTAD0930BO1&gclid=Cj0KCQiA_JTUBRD4ARIsAL7_VeXOQT1TCYHJZb7ifQDAGqbTm2uSwLsNB0_bR7o0pTSXHwROSrbpsSUaAtxPEALw_wcB&gclsrc=aw.ds&dclid=CLC_ypvEqNkCFYzdwAod8RYOVg | |
| 18:19:58 | mriedem | oh there is one on amazon now, well goes to show demand is up since last i checked | |
| 18:20:43 | mriedem | jaypipes: even have one for you https://ih0.redbubble.net/image.475009417.6220/flat,800x800,070,f.u5.jpg | |
| 18:28:34 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add require_tenant_aggregate request filter https://review.openstack.org/545002 | |
| 18:28:35 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add a require_tenant_trait request filter https://review.openstack.org/545079 | |
| 18:28:55 | dansmith | efried: ^ example of using traits to do the tenant isolation as an alternative | |
| 18:29:06 | efried | dansmith: Nice, looking. | |
| 18:29:21 | dansmith | efried: and a question in there you can probably answer and save me a few minutes of reading | |
| 18:30:50 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Detach volumes when VM creation fails https://review.openstack.org/544143 | |
| 18:33:49 | melwitt | efried: ack, np | |
| 18:36:01 | melwitt | mriedem: yep, good point to avoid nova-net and cells v1 for mox -> mock | |
| 18:38:40 | efried | dansmith: Responded | |
| 18:40:16 | dansmith | efried: tanks | |
| 18:42:09 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Clean up volumes on boot failure https://review.openstack.org/545086 | |
| 18:42:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Detach volumes when VM creation fails https://review.openstack.org/545087 | |
| 18:42:44 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add require_tenant_trait request filter https://review.openstack.org/545079 | |
| 18:42:49 | TheJulia | dansmith: so we managed to get nova not to crash in ironic's grenade jobs.... we had to inject -B on to the python command line to prevent the .pyc files from being placed on disk | |
| 18:43:06 | dansmith | TheJulia: I heard.. that sounds a lot like a python bug to me | |
| 18:43:24 | dansmith | TheJulia: it's one thing if we get some broken call and an exception or something, but a segv seems way out of the realm of reasonable to me | |
| 18:43:46 | TheJulia | dansmith: I believe it is officially a feature.... | |
| 18:43:54 | dansmith | TheJulia: ... | |
| 18:44:42 | TheJulia | certian app toolsets allow dynamic recompliation/reloading of python code in the app during runtime, the trick afaik is to remove the .pyc file which was likely occuring during upgrade | |
| 18:44:49 | TheJulia | regardless, we're hunting something breaking with placement | |
| 18:45:14 | dansmith | TheJulia: right, but that can't cause a segv and not be called a bug, IMHO | |
| 18:45:29 | efried | dansmith: The request_spec.project_id is always a UUID, yes? | |
| 18:45:36 | dansmith | TheJulia: dynamic recompile is cool, even if it causes some python call imcompatibility or something, but not a segv | |
| 18:45:47 | dansmith | efried: I think it depends on your keystone backend, no? | |
| 18:45:58 | dansmith | TheJulia: what placement thing are you chasing now? | |
| 18:46:38 | dansmith | efried: I bet lbragstad knows | |
| 18:47:01 | dansmith | efried: https://github.com/midokura/python-midonetclient/issues/19 | |
| 18:47:22 | efried | dansmith: no idea, swhy I'm asking. Cause it's gonna make a difference how much you have to sanitize it, etc. | |
| 18:47:22 | efried | But | |
| 18:47:22 | efried | I think we may be barking up the wrong tree anyway. Don't we actually want the trait to be CUSTOM_HOST_AGGREGATE_{agg_id} ? | |
| 18:47:27 | lbragstad | efried it depends on the resource backend being used | |
| 18:47:58 | TheJulia | dansmith: http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-placement-api.txt.gz#_Feb_15_18_00_31_109054 after everything gets up and running and nova-compute is able to post data back out, we're getting a conflict, but I'm afraid we don't understand the mechanisms in that publishing/use of data | |
| 18:47:58 | dansmith | efried: I don't :) | |
| 18:48:14 | TheJulia | dansmith: to then go backwards and figure out what is truly causing that failure | |
| 18:48:47 | dansmith | TheJulia: hmm, that's interesting | |
| 18:48:50 | dansmith | jaypipes: ^ | |
| 18:49:13 | lbragstad | efried if keystone is told to pull projects from something other than the default sql backend, then we can't guarantee them to be uuids | |
| 18:49:23 | dansmith | TheJulia: I'm not sure that's really a blocking thing.. is there a matching failure in the n-cpu log? | |
| 18:49:32 | TheJulia | http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-n-cpu.txt.gz#_Feb_15_18_00_31_112744 | |
| 18:49:48 | efried | lbragstad: Thanks. | |
| 18:50:12 | lbragstad | efried yep | |
| 18:50:44 | dansmith | TheJulia: okay that almost maybe kinda looks like someone changed ids or there's some confusion going on | |
| 18:50:58 | dansmith | TheJulia: multiple nova-computes? multiple ironic nodes? | |
| 18:52:14 | dansmith | actually, efried ^ | |
| 18:52:15 | TheJulia | dansmith: 2x n-cpu running pike, 2x ironic-conductor (1 master, 1 queens) 1x ironic api running queens. | |
| 18:52:30 | jroll | and multiple ironic nodes | |