| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-26 | |||
| 13:42:58 | sfinucan | So long as it's consistently correct :) | |
| 13:43:38 | johnthetubaguy | I am OK with consistently wrong, over in-consistently wrong, but maybe I am in a funny mood | |
| 13:45:41 | cdent | bauzas, johnthetubaguy so looking at that grenade run, it appears that placement was not configured on the old side. Was the change in option 4 supposed to cause that? | |
| 13:46:56 | bauzas | cdent: I'm writing an email | |
| 13:46:59 | bauzas | cdent: you'll see my thoughts | |
| 13:47:00 | cdent | k | |
| 13:47:15 | bauzas | after thinking more of that, I think we hit the bigger problem that johnthetubaguy said | |
| 13:47:33 | johnthetubaguy | I think its time for option 3 | |
| 13:47:35 | bauzas | which is that we need to ask operators to modify their compute configuration before upgrading | |
| 13:47:58 | johnthetubaguy | but if we limit it only for during an upgrade, that feels better than raw option 3, like option 3.11 or something | |
| 13:48:03 | bauzas | johnthetubaguy: could you please reply to my email I'm sending in 2 mins ? | |
| 13:48:25 | johnthetubaguy | bauzas: sure | |
| 13:49:26 | cdent | I wish I understood why it is hard to set the config on the subnode | |
| 13:50:33 | bauzas | johnthetubaguy: cdent: so I just sent my email | |
| 13:50:46 | bauzas | tl;dr: I think we have 2 problems, one being larger than other | |
| 13:51:17 | bauzas | the fact that if you force a destination it fails is to me a corner case that can easily be tackled if we make the scheduler more defensive | |
| 13:51:25 | bauzas | but | |
| 13:52:12 | bauzas | it raises the main issue that it requires nova.conf to be updated for all your computes before you upgrade to Ocata, and that's a larger action item than the one we stated in the upgrade process http://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process | |
| 13:52:26 | bauzas | where we only ask for modifying the upgrade_levels flag | |
| 13:52:51 | bauzas | and even that | |
| 13:53:30 | bauzas | cdent: johnthetubaguywe all supposed that a rolling upgrade means that you can only focus on upgrading your controller without touching the computes (unless just upgrading them to the latest stable release) | |
| 13:53:34 | bauzas | johnthetubaguy: $ | |
| 13:53:46 | bauzas | snap, I'm wearing gloves | |
| 13:53:55 | bauzas | that's cold here, but not a reason | |
| 13:54:58 | cdent | We've known for a few months that there will be config change on the compute nodes, it's part of why we made the upgrade ready script, isn't it? | |
| 13:56:23 | bauzas | cdent: tbh, I never identified that it would require a nova.conf change on old computes | |
| 13:57:01 | edleafe | bauzas: what change is required? | |
| 13:57:03 | cdent | we've said for a long time that newton computes will need to run placement before the ocata upgrade | |
| 13:57:20 | cdent | in order ot set inventory | |
| 13:57:33 | cdent | and in order to set inventory you have to have the [placement] section in nova.conf | |
| 13:57:55 | cdent | edleafe: it contains the auth bits to do placement service discovery | |
| 13:57:56 | bauzas | edleafe: what cdent just said | |
| 13:58:48 | bauzas | cdent: are we sure we are going to ask operators to modify all their computes nova.conf before upgrading ? | |
| 13:58:55 | edleafe | cdent: but isn't that what "have to run the placement service" would entail? | |
| 13:59:22 | cdent | edleafe: yes, exactly and thus bauzas: yes! | |
| 13:59:28 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient: [cli] Use environment variable for token https://review.openstack.org/425647 | |
| 14:00:10 | cdent | bauzas: we act like this is a big deal because we pretend like someone, the day after feature freeze is going to upgrade to ocata, but that's rarely the case | |
| 14:00:23 | cdent | people will be upgrading to newton while we are nearly finished fixing bugs in ocata | |
| 14:00:24 | edleafe | cdent: ok, that's what I thought. I felt I was missing bauzas concern | |
| 14:01:02 | edleafe | So... nova meeting? | |
| 14:01:21 | bauzas | I dunno | |
| 14:01:28 | bauzas | in theory yes | |
| 14:01:36 | bauzas | lemme see if there is an agenda | |
| 14:02:06 | johnthetubaguy | I vote we skip the meeting, seems like we are having it here | |
| 14:02:19 | cdent | johnthetubaguy: makes good sense given the day it is | |
| 14:02:44 | edleafe | no argument from me | |
| 14:03:18 | bauzas | so the agenda is not up to date | |
| 14:03:34 | bauzas | I could just open the meeting, do a vote for skipping it and leave us coming back here ? | |
| 14:04:02 | edleafe | bauzas: go for it | |
| 14:04:14 | sfinucan | aye, I vote skip | |
| 14:10:29 | openstackgerrit | Wang Qing wu proposed openstack/nova: WIP(6): Introduce base PowerVM driver (ssp drv) https://review.openstack.org/422594 | |
| 14:11:07 | johnthetubaguy | bauzas: I replied to your email | |
| 14:11:16 | bauzas | johnthetubaguy: ack | |
| 14:21:53 | claudiub|2 | sfinucan: ehm, it seems moshele isn't online atm, so can you take a look at the hyper-v pci passthrough patch and see if my answers seem reasonable to you? Thanks! https://review.openstack.org/#/c/420614/7 | |
| 14:29:50 | dansmith | johnthetubaguy: only /you/ can send this to its happy place: https://review.openstack.org/#/c/319379/ | |
| 14:30:16 | johnthetubaguy | dansmith: looking | |
| 14:30:52 | sfinucan | claudiub|2: Sure thing | |
| 14:32:38 | bauzas | the gate is 3 hours late, not a big deal | |
| 14:32:46 | bauzas | for a FF day :p | |
| 14:33:44 | bauzas | dansmith: I'm maybe particularly exhausted and short-sighted but have we agreed to ask operators to modify their nova.conf to amend placement section in newton before upgrading ? | |
| 14:34:06 | dansmith | bauzas: didn't we agree to that like weeks ago? | |
| 14:34:23 | bauzas | I think I'm nervous | |
| 14:34:24 | dansmith | if not, I'm not sure why we're doing all of this :) | |
| 14:34:49 | bauzas | and I think my brain is litterally getting me mad | |
| 14:35:26 | dansmith | I'm nervous that we might have to wait until pike to deliver any measurable scheduler improvement | |
| 14:35:42 | dansmith | by the way, | |
| 14:36:47 | dansmith | cdent: I know jaypipes did some scheduler benchmarking early on to test the sql-based approach, but do you know if anyone has measured the performance of the potentially large http queries the scheduler will be making? Like, under load with a large cloud... | |
| 14:36:59 | dansmith | I was thinking about that last night | |
| 14:37:08 | cdent | dansmith: have you seen the mail I sent last night on a similar topic | |
| 14:37:08 | claudiub|2 | dansmith: does this help? 2017-01-26 03:04:14.169 2780 WARNING nova.scheduler.client.report [req-b2db1cb5-b647-4e48-b939-b0dfd7509e17 - - - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading. | |
| 14:37:27 | dansmith | claudiub|2: help what? | |
| 14:37:35 | bauzas | dansmith: nothing really yet AFAIU | |
| 14:37:53 | bauzas | we haven't tested yet the performance, only what yingxin and jay did | |
| 14:38:07 | claudiub|2 | a, sorry, bauzas asked about the placement section in nova.conf. my bad. :) | |
| 14:38:09 | cdent | dansmith: my calculations are ~250 requests per second on a 1000 compute node cluster with 10 instances per node | |
| 14:38:21 | cdent | that vast majority of those GETs | |
| 14:38:21 | dansmith | claudiub|2: ah, yeah, that.. we added that warning to help exactly that, yeah :) | |
| 14:38:33 | johnthetubaguy | dansmith: are you very against doing scheduling the old way, but only until all compute nodes are upgraded? (a modified option 3) | |
| 14:38:38 | bauzas | okay, we need to move on | |
| 14:38:47 | cdent | and many of them redudant, with a few minor fixes to the resource_tracker (all in the email dansmith) | |
| 14:38:52 | dansmith | cdent: ah, okay, that's a pretty high rate right? | |
| 14:38:54 | dansmith | cdent: yeah, looking now | |
| 14:38:57 | cdent | dansmith: yes | |
| 14:39:00 | bauzas | I'll at least work on trying to have the force_hosts thing to accept an empty list | |
| 14:39:06 | johnthetubaguy | is that without any builds happening? | |
| 14:39:07 | dansmith | johnthetubaguy: I have been yeah | |
| 14:39:24 | cdent | johnthetubaguy: yes, that's after the builds are done and change has frozen | |
| 14:39:37 | johnthetubaguy | dansmith: don't get me wrong, I would love to do a clean switch, but I would rather that than not land it | |
| 14:39:42 | cdent | but it can be cut _way_ back with a few changes. halfed with a bug fix that's already in progress | |
| 14:40:05 | johnthetubaguy | cdent: that sounds like a worthy aim | |
| 14:40:44 | cdent | in order to make one fix, I/someone needs to understand why tracked_instances in the resource tracker is being cleared on every periodic job | |
| 14:40:46 | dansmith | cdent: okay, but also, we can scale that horizontally | |
| 14:41:02 | cdent | if that clear() doesn't happen, the number of requests can drop, alot | |
| 14:41:14 | cdent | dansmith: yeah, which is nicely easy | |
| 14:41:17 | dansmith | johnthetubaguy: the problem with that is that I think people will rely on that crutch | |
| 14:41:30 | cdent | yeah, that crutch is something we've been wanting to avoid | |
| 14:41:34 | dansmith | johnthetubaguy: but, if we check service_version and only do the fallback if there are old nodes, maybe that's a thing | |
| 14:41:45 | johnthetubaguy | dansmith: thats the one I was meaning | |
| 14:41:50 | dansmith | johnthetubaguy: that way once they've updated everything then they're stuck | |
| 14:41:51 | johnthetubaguy | dansmith: only during upgrade, allow the fallback | |