| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-16 | |||
| 21:03:31 | delete | GlanceConnectionFailed: Connection to glance host 198.23.82.11:9292 failed: Error finding address for https://198.23.82.11:9292/v1/images/detail?is_public=none&limit=20: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) | |
| 21:03:37 | delete | glance is using a self-signd certificate | |
| 21:03:52 | edleafe | bauzas: and I told you that I had done it in a subsequent patch | |
| 21:03:54 | bauzas | dansmith: well, I pinged you and edleafe this morning explaining why I wasn't yet uploading my series | |
| 21:05:09 | dansmith | bauzas: yeah, but I hadn't seen any updates since then.. I guess I missed them | |
| 21:05:13 | delete | nevermind | |
| 21:07:37 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Delete existing allocations when retrying https://review.openstack.org/460433 | |
| 21:07:37 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Don't delete allocation if instance being scheduled https://review.openstack.org/460177 | |
| 21:07:37 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Allocate instance by the conductor https://review.openstack.org/460455 | |
| 21:07:38 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Scheduler returning more than 1 host https://review.openstack.org/465175 | |
| 21:07:46 | bauzas | dansmith: so ^ | |
| 21:09:35 | dansmith | bauzas: cool, thanks.. I'm assuming edleafe pushed up something that caused a merge conflict? | |
| 21:10:05 | bauzas | FWIW, I'm just adding the RPC API version | |
| 21:16:07 | efried | dansmith Is there a generic, forward- and backward-compatible way I can tell (in my CI) whether cellsv2 is enabled? | |
| 21:16:18 | efried | Basically, I need to know whether to run that nova-manage command. | |
| 21:16:27 | dansmith | um | |
| 21:16:35 | dansmith | as of ocata it's always enabled | |
| 21:16:38 | dansmith | so, yes? :) | |
| 21:16:46 | efried | But would like to code it in such a way that my CI will work today on pre-ocata, but also into the future. | |
| 21:18:06 | dansmith | your CI doesn't know what release you're on? | |
| 21:18:33 | efried | dansmith Well, yeah, but it's kinda tough to say "greater than or equal to ocata" | |
| 21:19:00 | efried | I mean, that'll work (cause 'master' is >= 'ocata')... until we roll over from z back to a ;-) | |
| 21:19:35 | dansmith | welp... I dunno, maybe pick something you know is in ocata but not in newton that you can key off of? like the presence of cmd/nova-status or something? | |
| 21:20:26 | efried | Okay. I think we're going to go dumber than that. We're not CIing before newton, so we'll just say "if not newton". Thanks dansmith. | |
| 21:20:37 | dansmith | seems reasonable :) | |
| 21:32:09 | edleafe | bauzas: dansmith: I have another patch that adds returning multiple hosts to the conductor, but wasn't sure if there was an agreement on a) how to do it and b) how many | |
| 21:33:40 | dansmith | edleafe: should be num_retries or whatever | |
| 21:33:53 | dansmith | well, num_instances * num_retries | |
| 21:33:58 | dansmith | edleafe: there's another wrinkle | |
| 21:33:59 | edleafe | "whatever" :) | |
| 21:34:19 | dansmith | the hosts we slap into a reqspec to pass down to the cell all have to be ... in that cell | |
| 21:34:46 | dansmith | so I was thinking as a first go, we might just get the normal candidate hosts, and then for each one, ask for n-1 more that are in the same cell, since we can ask the scheduler that already | |
| 21:34:56 | dansmith | (for the same cell I mean) | |
| 21:35:19 | edleafe | so... the scheduler picks a host, and then only returns other hosts that are in the same cell? | |
| 21:35:25 | edleafe | IOW, a post-filter? | |
| 21:37:16 | dansmith | well, we pick num_instances hosts like normal, | |
| 21:37:35 | dansmith | but then for each instance we re-ask for num_retries host in the same cell | |
| 21:39:28 | edleafe | ok, so check the cell_uuid for each selected host, and only return more hosts with that same cell_uuid | |
| 21:40:06 | dansmith | yeah, we can already ask to limit hosts to a given cell_uuid, | |
| 21:40:32 | dansmith | so I guess we have to look up the host_mapping for that host so we know what cell_uuid to ask for, | |
| 21:40:49 | dansmith | but we have to do that somewhere in that code so we should be able to collapse those (eventually, if nothing else) | |
| 21:41:09 | edleafe | yeah, the order of events was not clear to me | |
| 21:41:17 | dansmith | but clear now? | |
| 21:41:37 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Scheduler returning more than 1 host https://review.openstack.org/465175 | |
| 21:41:37 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Allocate instance by the conductor https://review.openstack.org/460455 | |
| 21:41:38 | dansmith | I haven't looked at bauzas' code from just now, it's possible he's doing some of that | |
| 21:41:52 | bauzas | just updated my change | |
| 21:42:19 | bauzas | edleafe: did that in https://review.openstack.org/465175 given the consensus we had | |
| 21:42:30 | edleafe | dansmith: clear except for the request for a particular cell | |
| 21:42:38 | bauzas | one of the reasons why it took me a while to upload the series | |
| 21:43:03 | dansmith | edleafe: there's code in the migrate path that demonstrates that, but reqspec.destination.cell can be set, which limits it to that cel | |
| 21:43:20 | dansmith | bauzas: okay cool | |
| 21:43:38 | edleafe | bauzas: ok, thanks. I was looking for getting the cell uuid from the reqspec | |
| 21:44:16 | dansmith | I need to run off for a bit, but I will look over bauzas' patches when I get back | |
| 21:44:33 | edleafe | And was it agreed to return a 2-tuple of (chosen_host, alternative_hosts), or simply a list of hosts, with the first one the chosen one? | |
| 21:45:22 | bauzas | edleafe: don't forget that the scheduler can return more than 1 | |
| 21:45:35 | dansmith | edleafe: we can't really do it all in one query because of the cell restriction I think | |
| 21:45:37 | bauzas | hint : num_instances | |
| 21:46:40 | dansmith | anyway, biab | |
| 21:46:44 | edleafe | bauzas: yeah, of course I get that. We were originally returning a list of single hosts; we could now return a list of lists of hosts | |
| 21:46:47 | dansmith | don't kill each other while I'm gone | |
| 21:47:01 | edleafe | dansmith: you're no fun | |
| 21:47:09 | bauzas | it's 23:47 | |
| 21:47:16 | bauzas | nothing good happens after midnight | |
| 21:47:17 | dansmith | okay fine, do and I'll congratulate the winner when I return | |
| 21:47:18 | melwitt | what bauzas has at https://review.openstack.org/#/c/465175/2/nova/scheduler/filter_scheduler.py@142 is what I was thinking, without doing a second query | |
| 21:47:43 | melwitt | that is, parse the list we get back the first time to pick alternatives in the same cell | |
| 21:48:05 | dansmith | oh because we're getting back more than subset size now | |
| 21:48:19 | melwitt | we always get the whole list I thought (all that pass all filters) | |
| 21:48:29 | edleafe | melwitt: that's correct | |
| 21:48:31 | melwitt | and then the first is picked unless subset size is bigger than 1 | |
| 21:48:40 | dansmith | we return 10k hosts from scheduler to conductor? | |
| 21:48:54 | edleafe | dansmith: no | |
| 21:49:02 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Scheduler returning alternatives per destination https://review.openstack.org/465175 | |
| 21:49:02 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Allocate instance by the conductor https://review.openstack.org/460455 | |
| 21:49:10 | edleafe | we return 10K hosts from placement to scheduler (potentially) | |
| 21:49:14 | bauzas | I stop iterating over the list once we have enough alternatives | |
| 21:49:19 | bauzas | but please see $code | |
| 21:49:20 | dansmith | oh, I see that's happening in scheduler, yeah, okay | |
| 21:49:26 | melwitt | yeah | |
| 21:49:31 | bauzas | seriously, the big problem is not there | |
| 21:49:35 | dansmith | yeah I didn't think about doing that, but that's good | |
| 21:49:51 | bauzas | like I said yesterday in that channel, we stop using iterators | |
| 21:50:02 | melwitt | bauzas: yeah, that's exactly what I hoped to see, stop when have enough instead of doing a list comprehension over all | |
| 21:50:07 | dansmith | I guess I thought we weighted a subset of the hosts, | |
| 21:50:13 | bauzas | so in case we have 10K nodes, then we have a 10K list for filters and a 10K list for weighers | |
| 21:50:21 | dansmith | because on early scheduler runs, we'll be weighing 10k nodes just to pick one | |
| 21:50:28 | bauzas | instead of having iterators | |
| 21:50:42 | bauzas | so it took me 3 hours to see if I was able to quickly fix that | |
| 21:50:58 | bauzas | ie. using iterators for filters and weighers | |
| 21:51:08 | bauzas | looks to me it's possible, but I need more time | |
| 21:51:56 | dansmith | not sure how you can pick the highest score host without iterating all of them... | |
| 21:51:59 | dansmith | if that's what you mean | |
| 21:52:51 | bauzas | dansmith: I was thinking of adding a reservoir sampling method for that | |
| 21:52:52 | edleafe | filters have to iterate through all returned hosts. Weighers have to iterate through all filtered hosts | |
| 21:53:05 | dansmith | edleafe: right | |
| 21:53:14 | bauzas | we need to iterate for sure, but we don't need to put it all in memory | |
| 21:53:22 | dansmith | sure | |
| 21:53:35 | dansmith | anyway, I'll look more at this when I get back | |
| 21:54:24 | bauzas | so, for example, to-be-weighting hosts could be like having an index and a weight so we could just sort based on that | |