| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 12:22:03 | jroll | true | |
| 12:22:27 | johnthetubaguy | well, feels liek we should add some user concepts in, when we can't assume that | |
| 12:23:08 | jroll | maybe the reno is just "if you get build failures trying to schedule to a non-existent node, check *waves hands* stuff, you might need to delete the RP" | |
| 12:23:13 | cdent | I gotta break away for several minutes, perhaps we can all percolate on this for a small window of time and we'll know something after that? | |
| 12:23:24 | cdent | jroll++ :) | |
| 12:24:58 | johnthetubaguy | yeah, a vague reno is better than no reno or fix getting in | |
| 12:25:26 | cdent | bbs | |
| 12:28:21 | BobBall | johnthetubaguy / mriedem: Citrix XenServer CI is back healthy now - can username citrix_xenserver_ci be re-added to the voting group https://review.openstack.org/#/admin/groups/511,members? | |
| 13:06:42 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove redundant bdm serial mangling and saving during swap_volume https://review.openstack.org/427364 | |
| 13:18:02 | johnthetubaguy | BobBall: OK | |
| 13:18:42 | openstackgerrit | Silvan Kaiser proposed openstack/nova master: Removes getfattr from Quobyte Nova driver https://review.openstack.org/428646 | |
| 13:18:59 | johnthetubaguy | BobBall: is a vote on this one coming really soon? https://review.openstack.org/#/c/424428/ | |
| 13:19:34 | BobBall | johnthetubaguy: If by soon you mean 25 minutes ago, then yes | |
| 13:19:53 | BobBall | We commented but didn't having voting rights so no vote :) | |
| 13:25:31 | tdurakov | BobBall: in case of re-roll please add change-log comment to xen migrate data object | |
| 13:26:06 | johnthetubaguy | BobBall: heh, thats soon, cool. turns out you have to click refresh | |
| 13:26:31 | BobBall | tdurakov: Sure will do - what do you mean by re-roll though? | |
| 13:26:39 | tdurakov | BobBall: also, what if I'm migration instances, from 6.5 to 7.0 or vice versa is this field required for both of them? | |
| 13:27:01 | tdurakov | BobBall: resubmit patch | |
| 13:27:08 | tdurakov | in case of ^ | |
| 13:28:05 | johnthetubaguy | tdurakov: I believe this is Nova version related, not XenServer version related | |
| 13:28:07 | BobBall | tdurakov: Migrating from 7.0 to 6.5 isn't supported by XenServer - XAPI will prevent that explicitly. 6.5 to 7.0 will work as the source doesn't look at or use the vif_map, it's prepared by and used by the target. | |
| 13:28:42 | tdurakov | johnthetubaguy: from the but definiton I understood that xen version affect this as well | |
| 13:28:51 | tdurakov | s/but/bug | |
| 13:28:57 | johnthetubaguy | yeah, I see that detail now | |
| 13:28:59 | johnthetubaguy | oops | |
| 13:29:00 | mriedem | cdent: johnthetubaguy: jroll: vdrok: edleafe: on https://review.openstack.org/#/c/428375/ i think we actually should raise a ResourceProviderInUse or some kind of exception if we delete the compute node but can't delete the resource provider, | |
| 13:29:05 | BobBall | johnthetubaguy: XS 7.0 introduced the requirement of the vif map | |
| 13:29:06 | mriedem | because that means we won't reschedule | |
| 13:29:44 | cdent | mriedem: but what would catch it? it's already getting logged as a warning? | |
| 13:30:00 | mriedem | way up in the compute manager | |
| 13:30:03 | johnthetubaguy | mriedem: I thought this was just getting called from some periodic task? | |
| 13:30:07 | mriedem | the thing that caught the NodeNotFound in the bug report | |
| 13:30:09 | mriedem | no | |
| 13:30:12 | mriedem | well, | |
| 13:30:18 | johnthetubaguy | both I guess | |
| 13:30:19 | mriedem | yeah it is | |
| 13:30:29 | mriedem | the periodic task doesn't delete the RP, | |
| 13:30:32 | mriedem | and then the scheduler uses it, | |
| 13:30:44 | mriedem | which then blows up with NodeNotFound in build_and_run_instances | |
| 13:30:48 | mriedem | which triggers a reschedule to another node | |
| 13:30:49 | mriedem | so, | |
| 13:30:56 | mriedem | i guess raising the exception up doesn't help | |
| 13:30:58 | mriedem | fooy | |
| 13:31:04 | cdent | indeed | |
| 13:31:30 | cdent | this is messy. mriedem did you see all the chatter above about how to word a reno note (I linked on the review if not) | |
| 13:31:44 | mriedem | i didn't read that yet | |
| 13:32:17 | mriedem | "btw this optional thing which you might be using might be borking your ironic scheduling but then again if you're not using it so nevermind but just in case you were and hit this thing....beware" | |
| 13:32:23 | mriedem | something like that? ^ | |
| 13:33:15 | cdent | well yeah, exactly, that's pretty much the problem | |
| 13:33:49 | cdent | I'm drafting something now | |
| 13:34:20 | johnthetubaguy | could we use that NodeNotFound error to trigger a check of that resource provider, thinking about it | |
| 13:34:36 | openstackgerrit | Sergey Nikitin proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 13:34:40 | vdrok | if the compute node was deleted but resource provider was not, the bit of deleting resource provider will not be retried by the periodic task? | |
| 13:36:38 | jroll | vdrok: yeah, that could be a problem, maybe we do it the other way (RP first, CN second), and if RP blows up don't delete the compute node either | |
| 13:36:44 | jroll | hrm | |
| 13:37:17 | cdent | mriedem, johnthetubaguy, vdrok, jroll: I'll hold off on pushing up anything until this discussion resolves in some fashion, otherwise it feels like we're going to churn | |
| 13:37:17 | jroll | johnthetubaguy: oh, we totally could | |
| 13:37:46 | jroll | cdent: maybe etherpad the reno draft for now, then, so we can get that handled? | |
| 13:37:57 | johnthetubaguy | mriedem: I am tempted to merge the current fix with a note, that stops *more* badness when stuff gets deleted, feels like we discuss the extra bits on top? | |
| 13:38:20 | cdent | jroll: except that it sounds like we potentially have a more robust fix in the works, which would change the note quite a bit? | |
| 13:38:26 | mriedem | johnthetubaguy: yeah i'm not looking to get a comprehensive fix in for rc1 | |
| 13:38:32 | cdent | or that, okay | |
| 13:38:35 | jroll | heh | |
| 13:38:41 | mriedem | we can backport something if needed | |
| 13:38:45 | johnthetubaguy | +1 | |
| 13:38:55 | mriedem | honestly i doubt very much that anyone is using placement yet, if even on newton yet | |
| 13:39:28 | johnthetubaguy | I think thats fair, based on the massive bugs we found in the last few days | |
| 13:39:53 | jroll | yeah, so to trigger this, you need newton running with placement, with ironic, and delete at least one ironic node | |
| 13:40:01 | jroll | and it will be triggered upon install of current master | |
| 13:40:05 | jroll | right? | |
| 13:41:14 | mriedem | not just install master, | |
| 13:41:20 | mriedem | but bring all of your computes up to ocata | |
| 13:41:28 | mriedem | so full upgrade of all nova services | |
| 13:42:05 | jroll | I don't think so | |
| 13:42:13 | jroll | the bug is in newton if you're running placement | |
| 13:42:25 | jroll | because you have inventories for things that don't exist | |
| 13:42:38 | mriedem | jroll: but the scheduler in newton doesn't use placement | |
| 13:42:38 | jroll | oh wait, the scheduler in ocata checks compute version, doesn't it | |
| 13:43:25 | sean-k-mooney | mriedem: if you have got a moment would you have any objection to me creating an experimatal job for nova to run a subset of what is currently run in the intel NFV ci? we will be moving where the NFV ci is hosted in then next 2-4 months and now seems like a good opurtnity to upstream the tests that dont require special hardware. those that do will remain in the nfv ci. | |
| 13:43:27 | jroll | the scheduler in ocata does, though, I just don't know if it waits for computes to be upgraded before using placement | |
| 13:44:22 | cdent | jroll: it does | |
| 13:44:23 | cdent | recent change | |
| 13:44:44 | jroll | ok cool, so builds don't break until you upgrade everything to ocata | |
| 13:44:55 | jroll | thanks cdent | |
| 13:46:08 | cdent | here's a stab at the reno: https://etherpad.openstack.org/p/bug-1661258-reno | |
| 13:46:13 | cdent | rather hand wavey | |
| 13:47:08 | cdent | or should it be an issue instead of fixes? | |
| 13:47:24 | jroll | maybe both? | |
| 13:48:03 | mriedem | sean-k-mooney: sure | |
| 13:48:34 | mriedem | cdent: jroll: i'd just have 'issues' as in known issue | |
| 13:48:54 | cdent | is leading to a reschedule the only outcome? I thought it could be more dire | |
| 13:50:02 | mriedem | it triggers a reschedule | |
| 13:50:06 | jroll | hm, is there any way to trigger this for not ironic nodes? | |
| 13:50:09 | mriedem | of which you get 3 and then NoValidHost | |
| 13:50:29 | mriedem | jroll: you'd have to delete the compute node records manually for non-ironic | |
| 13:51:16 | jroll | mriedem: right, because n-cpu wouldn't be running for a hyp that went away right | |
| 13:51:36 | jroll | s/went away/was thrown in the dumpster/ | |
| 13:51:43 | mriedem | n-cpu can be running and if it is, it will recreate the compute node record in the db | |
| 13:51:47 | mriedem | but the service record might be gone | |