| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 13:29:06 | mriedem | because that means we won't reschedule | |
| 13:29:44 | cdent | mriedem: but what would catch it? it's already getting logged as a warning? | |
| 13:30:00 | mriedem | way up in the compute manager | |
| 13:30:03 | johnthetubaguy | mriedem: I thought this was just getting called from some periodic task? | |
| 13:30:07 | mriedem | the thing that caught the NodeNotFound in the bug report | |
| 13:30:09 | mriedem | no | |
| 13:30:12 | mriedem | well, | |
| 13:30:18 | johnthetubaguy | both I guess | |
| 13:30:19 | mriedem | yeah it is | |
| 13:30:29 | mriedem | the periodic task doesn't delete the RP, | |
| 13:30:32 | mriedem | and then the scheduler uses it, | |
| 13:30:44 | mriedem | which then blows up with NodeNotFound in build_and_run_instances | |
| 13:30:48 | mriedem | which triggers a reschedule to another node | |
| 13:30:49 | mriedem | so, | |
| 13:30:56 | mriedem | i guess raising the exception up doesn't help | |
| 13:30:58 | mriedem | fooy | |
| 13:31:04 | cdent | indeed | |
| 13:31:30 | cdent | this is messy. mriedem did you see all the chatter above about how to word a reno note (I linked on the review if not) | |
| 13:31:44 | mriedem | i didn't read that yet | |
| 13:32:17 | mriedem | "btw this optional thing which you might be using might be borking your ironic scheduling but then again if you're not using it so nevermind but just in case you were and hit this thing....beware" | |
| 13:32:23 | mriedem | something like that? ^ | |
| 13:33:15 | cdent | well yeah, exactly, that's pretty much the problem | |
| 13:33:49 | cdent | I'm drafting something now | |
| 13:34:20 | johnthetubaguy | could we use that NodeNotFound error to trigger a check of that resource provider, thinking about it | |
| 13:34:36 | openstackgerrit | Sergey Nikitin proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 13:34:40 | vdrok | if the compute node was deleted but resource provider was not, the bit of deleting resource provider will not be retried by the periodic task? | |
| 13:36:38 | jroll | vdrok: yeah, that could be a problem, maybe we do it the other way (RP first, CN second), and if RP blows up don't delete the compute node either | |
| 13:36:44 | jroll | hrm | |
| 13:37:17 | jroll | johnthetubaguy: oh, we totally could | |
| 13:37:17 | cdent | mriedem, johnthetubaguy, vdrok, jroll: I'll hold off on pushing up anything until this discussion resolves in some fashion, otherwise it feels like we're going to churn | |
| 13:37:46 | jroll | cdent: maybe etherpad the reno draft for now, then, so we can get that handled? | |
| 13:37:57 | johnthetubaguy | mriedem: I am tempted to merge the current fix with a note, that stops *more* badness when stuff gets deleted, feels like we discuss the extra bits on top? | |
| 13:38:20 | cdent | jroll: except that it sounds like we potentially have a more robust fix in the works, which would change the note quite a bit? | |
| 13:38:26 | mriedem | johnthetubaguy: yeah i'm not looking to get a comprehensive fix in for rc1 | |
| 13:38:32 | cdent | or that, okay | |
| 13:38:35 | jroll | heh | |
| 13:38:41 | mriedem | we can backport something if needed | |
| 13:38:45 | johnthetubaguy | +1 | |
| 13:38:55 | mriedem | honestly i doubt very much that anyone is using placement yet, if even on newton yet | |
| 13:39:28 | johnthetubaguy | I think thats fair, based on the massive bugs we found in the last few days | |
| 13:39:53 | jroll | yeah, so to trigger this, you need newton running with placement, with ironic, and delete at least one ironic node | |
| 13:40:01 | jroll | and it will be triggered upon install of current master | |
| 13:40:05 | jroll | right? | |
| 13:41:14 | mriedem | not just install master, | |
| 13:41:20 | mriedem | but bring all of your computes up to ocata | |
| 13:41:28 | mriedem | so full upgrade of all nova services | |
| 13:42:05 | jroll | I don't think so | |
| 13:42:13 | jroll | the bug is in newton if you're running placement | |
| 13:42:25 | jroll | because you have inventories for things that don't exist | |
| 13:42:38 | jroll | oh wait, the scheduler in ocata checks compute version, doesn't it | |
| 13:42:38 | mriedem | jroll: but the scheduler in newton doesn't use placement | |
| 13:43:25 | sean-k-mooney | mriedem: if you have got a moment would you have any objection to me creating an experimatal job for nova to run a subset of what is currently run in the intel NFV ci? we will be moving where the NFV ci is hosted in then next 2-4 months and now seems like a good opurtnity to upstream the tests that dont require special hardware. those that do will remain in the nfv ci. | |
| 13:43:27 | jroll | the scheduler in ocata does, though, I just don't know if it waits for computes to be upgraded before using placement | |
| 13:44:22 | cdent | jroll: it does | |
| 13:44:23 | cdent | recent change | |
| 13:44:44 | jroll | ok cool, so builds don't break until you upgrade everything to ocata | |
| 13:44:55 | jroll | thanks cdent | |
| 13:46:08 | cdent | here's a stab at the reno: https://etherpad.openstack.org/p/bug-1661258-reno | |
| 13:46:13 | cdent | rather hand wavey | |
| 13:47:08 | cdent | or should it be an issue instead of fixes? | |
| 13:47:24 | jroll | maybe both? | |
| 13:48:03 | mriedem | sean-k-mooney: sure | |
| 13:48:34 | mriedem | cdent: jroll: i'd just have 'issues' as in known issue | |
| 13:48:54 | cdent | is leading to a reschedule the only outcome? I thought it could be more dire | |
| 13:50:02 | mriedem | it triggers a reschedule | |
| 13:50:06 | jroll | hm, is there any way to trigger this for not ironic nodes? | |
| 13:50:09 | mriedem | of which you get 3 and then NoValidHost | |
| 13:50:29 | mriedem | jroll: you'd have to delete the compute node records manually for non-ironic | |
| 13:51:16 | jroll | mriedem: right, because n-cpu wouldn't be running for a hyp that went away right | |
| 13:51:36 | jroll | s/went away/was thrown in the dumpster/ | |
| 13:51:43 | mriedem | n-cpu can be running and if it is, it will recreate the compute node record in the db | |
| 13:51:47 | mriedem | but the service record might be gone | |
| 13:51:54 | sean-k-mooney | mriedem: ok ill start working on the job. after a while i would hope we can move it to the check pipeline as non voting and then eventually voteing once it is shown to be stable. | |
| 13:52:26 | jroll | mriedem: right-o | |
| 13:53:20 | jroll | okay, I think I'm happy with that reno | |
| 13:53:28 | cdent | jroll, mriedem I we're happy with that content I can smooth it | |
| 13:53:31 | jroll | and I'm good with the fix as-is, for now, given the reno | |
| 13:53:37 | jroll | cdent: ++ | |
| 13:53:58 | sean-k-mooney | mriedem: im also going to work on some jobs for os-vif and a few other repos. i may have an update at the ptg on nfv testing in general. | |
| 13:54:53 | mriedem | sean-k-mooney: splendid | |
| 13:54:57 | mriedem | thanks for working on that | |
| 13:56:20 | sean-k-mooney | mriedem: well its been kindof buging me that we dont have a linux bridge job for os-vif there are some other gaps in the unit testing in os-vif and nova that i want to take a look at too to tighten up the testing a little | |
| 13:58:32 | openstackgerrit | Chris Dent proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 13:58:39 | cdent | ^^ adds the reno and changes one bug to partial-bug | |
| 13:58:45 | bauzas | mriedem: FWIW, saw your discussion with dansmith and melwitt yesterday about BFV instances | |
| 13:59:21 | jroll | cdent: trailing space on the reno, not sure if that breaks some check job | |
| 13:59:26 | bauzas | mriedem: so I saw your comment on gerrit and replied to it by saying that we should still verify the disk resource in placement | |
| 13:59:30 | cdent | dag nab it | |
| 13:59:36 | bauzas | mriedem: unless the disk value is 0 | |
| 14:00:02 | openstackgerrit | Chris Dent proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 14:00:03 | bauzas | mriedem: because once we will have Cinder RPs, then we will still take a allocation for the disk size | |
| 14:01:15 | jroll | cdent: +1'd. now I'm wondering if we want to catch NodeNotFound and have a better error message, but it's probably fine | |
| 14:01:50 | mriedem | jroll: you could still probably get the NodeNotFound for other reasons | |
| 14:01:58 | mriedem | like some kind of race to delete the node and schedule to it at the same time | |
| 14:02:01 | mriedem | i'd just leave it | |
| 14:02:18 | jroll | mriedem: good point | |
| 14:02:38 | mriedem | bauzas: i don't really have that all in my head, but dansmith has a WIP up for some of this | |
| 14:02:55 | mriedem | https://review.openstack.org/#/c/428481/ | |
| 14:07:55 | figleaf | cdent: thanks for picking up https://review.openstack.org/428375 | |
| 14:08:39 | mriedem | +2 from me | |