| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 10:43:17 | bauzas | problem solved. | |
| 10:43:22 | johnthetubaguy | indeed :) | |
| 10:43:26 | bauzas | anyway, I need to bail out, school time | |
| 10:43:55 | bauzas | I could pretend the ISS is still on Thursday given they cross the world every 1.5h | |
| 10:44:13 | johnthetubaguy | that might give us a few more hours, sweet | |
| 10:44:32 | bauzas | just ask Thomas Pesquet to pull the +W on the patch | |
| 10:46:32 | johnthetubaguy | lyarwood: are we sure our source is always the same format at what is listed in /proc/mounts | |
| 10:46:44 | johnthetubaguy | lyarwood: just wondering about nfs vs nfs3, etc | |
| 10:48:50 | lyarwood | johnthetubaguy: hmm I'm pretty sure they are the same but let me double check | |
| 10:49:21 | johnthetubaguy | I guess if it works for findmnt it kinda has to be, but just wanted to double check | |
| 10:55:23 | cdent | anything special I should poke at, or let the etherpad be my guide? | |
| 10:55:47 | johnthetubaguy | I think the etherpad is still king | |
| 10:56:04 | johnthetubaguy | at this point, seems like its looking for things to backport to RC2 | |
| 11:02:45 | cdent | i'll get the fixes for https://review.openstack.org/#/c/428375/ | |
| 11:06:02 | tdurakov | johnthetubaguy: got question on that one and xen-api(https://review.openstack.org/#/c/424428/) live-migration process - what is the common process upgrading xen server in openstack( 6.5 > 7.0) | |
| 11:06:43 | tdurakov | any recommendations on xenapi multinode deployment? | |
| 11:07:57 | johnthetubaguy | tdurakov: the way we used to do it was live migrate everything off the box, re-image box. I am sure you could do in place upgrade, but all the VMs will be turned off for the duration | |
| 11:08:39 | johnthetubaguy | tdurakov: also, never attempt to use the pools feature with Nova, just have several standalone boxes | |
| 11:08:55 | johnthetubaguy | tdurakov: but the expert these days is BobBall, he can help you through the details | |
| 11:09:25 | tdurakov | BobBall: are you around? | |
| 11:09:47 | tdurakov | johnthetubaguy: thanks for details | |
| 11:10:09 | johnthetubaguy | tdurakov: no worries | |
| 11:10:38 | tdurakov | johnthetubaguy: no ci for xen multinode, so it's kind of tough thing to reveiw | |
| 11:11:07 | johnthetubaguy | tdurakov: yeah, not sure if they have an internal one | |
| 11:12:42 | openstackgerrit | Bob Ball proposed openstack/nova master: Fix live migrate with XenServer https://review.openstack.org/424428 | |
| 11:13:10 | BobBall | hi tdurakov :) I am indeed around. | |
| 11:13:30 | tdurakov | BobBall: hi, want to make some testing of ^ | |
| 11:13:52 | BobBall | And no, no multinode CI. Current testing is done manually. We recently got some more hardware installed in our CI env though with the purpose of enabling multinode for live migrate testing. Hopefully that'll be added in the next few weeks. | |
| 11:14:21 | tdurakov | what's the best way doing multinode xen installation now? | |
| 11:14:23 | BobBall | To test I normally set up two normal devstack environments then connect them together | |
| 11:14:42 | tdurakov | ok | |
| 11:17:02 | lyarwood | johnthetubaguy: sorry took a while to ./stack.sh with the NFS plugin, both v3 and v4 leave the same source URI in /proc/mounts - http://paste.openstack.org/show/597531/ | |
| 11:17:39 | johnthetubaguy | lyarwood: oh cool, thanks for digging | |
| 11:18:27 | lyarwood | johnthetubaguy: np it was worth it tbh | |
| 11:22:19 | openstackgerrit | Chris Dent proposed openstack/nova master: Delete a compute node's resource provider when node is deleted https://review.openstack.org/428375 | |
| 11:24:57 | cdent | johnthetubaguy: responded to you about the 404 thing: it's if not 404, log | |
| 11:25:03 | cdent | so everything unusual is logged | |
| 11:25:17 | johnthetubaguy | cdent: balls, I read that as == | |
| 11:25:35 | cdent | on the bugger scale where do balls fit? | |
| 11:25:43 | johnthetubaguy | its hard to know, lol | |
| 11:26:36 | cdent | I made the exact same mistake where the same idiom was used elsewhere (I think on deleting allocations, but not sure). | |
| 11:26:56 | jroll | morning y'all | |
| 11:26:58 | cdent | johnthetubaguy: i put up a new version of that while you were commenting. I think getting all the previous comments | |
| 11:27:09 | johnthetubaguy | cdent: yeah, just taking a peak | |
| 11:27:10 | jroll | "It seems to be the webob pattern to make a non-2xx response False-y" webob whyyyy | |
| 11:27:14 | cdent | morning jroll, you might want to look at https://review.openstack.org/#/c/428375/ too | |
| 11:27:24 | cdent | that's request not webob doing that | |
| 11:27:25 | jroll | yes, I'm already there :) | |
| 11:27:36 | jroll | ok requests whyyyy | |
| 11:27:46 | jroll | :) | |
| 11:27:49 | cdent | and I would guess they do it because, like, man, requests is http for humans, man | |
| 11:28:34 | cdent | it vaguely makes sense, but it does make for some confusion, and is to me biased | |
| 11:28:56 | cdent | I think response object would only be falsey if the web server couldn't be reached | |
| 11:29:02 | jroll | yeah, it's non-obvious | |
| 11:29:09 | jroll | oh well, learned something new today | |
| 11:29:21 | johnthetubaguy | its better than success being falsey I guess | |
| 11:32:21 | andymccr | if anybody is about to give https://review.openstack.org/#/c/426163/ a review, that'd be really appreciated :) | |
| 11:34:38 | johnthetubaguy | cdent: should we worry about the compute nodes we already deleted, but have entries for in the placement API, I am thinking probably no? | |
| 11:38:49 | cdent | johnthetubaguy: I'm not sure how we'd figure that out in that direction. | |
| 11:39:35 | johnthetubaguy | cdent: good question, I guess we can't list the resource providers by host as such, only node? | |
| 11:40:54 | johnthetubaguy | cdent: I do wonder about a nova-manage tool to "re-sync" placement, and check for any odd things, its something that could live in such a tool. Although creating that tool feels a bit too much like giving up. | |
| 11:44:19 | jroll | johnthetubaguy: left a comment, I'm thinking we throw people a release note about that, at least? | |
| 11:44:28 | jroll | patch lgtm otherwise though | |
| 11:46:04 | cdent | johnthetubaguy: yeah the host, node stuff is not something that the placement side knows. it just knows resource providers. it's up to the "nova" side to do that jiggery pokery | |
| 11:47:42 | cdent | jroll: That release note seems potentially very confusing and scary for people and might lead to random deletions against the api? | |
| 11:48:31 | jroll | cdent: yeah, we can add big scary warnings or something | |
| 11:48:56 | jroll | warning: this only applies if you're doing CD by the day/week | |
| 11:48:59 | jroll | idk maybe it's fine | |
| 11:49:06 | jroll | feels like we should tell people somehow though | |
| 11:57:55 | johnthetubaguy | cdent: jroll: oh, so this is affecting ironic already, I kept thinking it wasn't | |
| 11:58:08 | johnthetubaguy | wires crossed in my head | |
| 11:58:11 | jroll | johnthetubaguy: yeah, since a few days ago, I believe | |
| 11:59:54 | johnthetubaguy | surely since last cycle when we started reporting the inventory? | |
| 12:00:14 | johnthetubaguy | or did we skip ironic till really recently? | |
| 12:00:40 | jroll | johnthetubaguy: oh. ohhhhhh. | |
| 12:00:46 | jroll | damn | |
| 12:00:57 | jroll | yeah, we only noticed it when scheduling switched to inventory | |
| 12:01:12 | johnthetubaguy | oh, right, its only impacting since we started caring about whats in there | |
| 12:01:13 | jroll | but I guess the real problem might be much longer eh | |
| 12:01:33 | jroll | I guess we could spin up a newton install and test this easily | |
| 12:02:51 | cdent | johnthetubaguy: are you taking typing lessons from me "I was forgetting this could already an issue in production for folks, we should do that thing." | |
| 12:03:15 | johnthetubaguy | heh | |
| 12:03:53 | cdent | So how should this reno express itself? Do we explain just the issue or the issue and the hack for how to deal with it? Or something else? | |
| 12:04:45 | johnthetubaguy | cdent: its probably worth saying the problem you will see with iroinc and placement if you don't clear out old deleted nodes | |
| 12:05:15 | johnthetubaguy | cdent: its tempting to hack up some tool to help fix that, but we good to stop the bleeding with this patch anyways | |
| 12:06:23 | cdent | Just so I'm clear on the situation how would that hack operate? How do you know which things on the placement api side to delete? | |
| 12:06:50 | johnthetubaguy | because they are not in the current compute node list? I am not 100% sure actually | |
| 12:07:16 | jroll | doesn't resource_provider.uuid == ironic_node.uuid here? | |
| 12:09:28 | johnthetubaguy | cdent: so compute node is soft_delete, so if they don't trim that table, we could double check from soft deleted compute nodes to double check they are all removed | |
| 12:12:11 | cdent | but if they do, then they truly are orphans, unless the rp.name is useful in some fashion? | |
| 12:14:13 | cdent | so the hack is for each get_all_soft_delete_compute_nodes; try_to_delete_rp_with_same_uuid() | |
| 12:14:32 | johnthetubaguy | name is hypervisor_hostname I think | |
| 12:14:49 | johnthetubaguy | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L630 | |
| 12:16:09 | jroll | and hypervisor_hostname is the ironic node uuid | |
| 12:16:30 | johnthetubaguy | can you get a list of the deleted uuids though? | |
| 12:16:46 | johnthetubaguy | I mean you can get all the current ones and spot the difference | |
| 12:16:50 | johnthetubaguy | maybe thats the best way | |
| 12:16:57 | jroll | yeah that'd be the only way | |
| 12:17:19 | cdent | blechy | |
| 12:17:32 | jroll | for cn in compute_nodes: if ironic.get_node(cn.hypervisor_hostname) gives a 404: deleted_nodes.append(cn.hypervisor_hostname) | |