| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-16 | |||
| 21:49:10 | edleafe | we return 10K hosts from placement to scheduler (potentially) | |
| 21:49:14 | bauzas | I stop iterating over the list once we have enough alternatives | |
| 21:49:19 | bauzas | but please see $code | |
| 21:49:20 | dansmith | oh, I see that's happening in scheduler, yeah, okay | |
| 21:49:26 | melwitt | yeah | |
| 21:49:31 | bauzas | seriously, the big problem is not there | |
| 21:49:35 | dansmith | yeah I didn't think about doing that, but that's good | |
| 21:49:51 | bauzas | like I said yesterday in that channel, we stop using iterators | |
| 21:50:02 | melwitt | bauzas: yeah, that's exactly what I hoped to see, stop when have enough instead of doing a list comprehension over all | |
| 21:50:07 | dansmith | I guess I thought we weighted a subset of the hosts, | |
| 21:50:13 | bauzas | so in case we have 10K nodes, then we have a 10K list for filters and a 10K list for weighers | |
| 21:50:21 | dansmith | because on early scheduler runs, we'll be weighing 10k nodes just to pick one | |
| 21:50:28 | bauzas | instead of having iterators | |
| 21:50:42 | bauzas | so it took me 3 hours to see if I was able to quickly fix that | |
| 21:50:58 | bauzas | ie. using iterators for filters and weighers | |
| 21:51:08 | bauzas | looks to me it's possible, but I need more time | |
| 21:51:56 | dansmith | not sure how you can pick the highest score host without iterating all of them... | |
| 21:51:59 | dansmith | if that's what you mean | |
| 21:52:51 | bauzas | dansmith: I was thinking of adding a reservoir sampling method for that | |
| 21:52:52 | edleafe | filters have to iterate through all returned hosts. Weighers have to iterate through all filtered hosts | |
| 21:53:05 | dansmith | edleafe: right | |
| 21:53:14 | bauzas | we need to iterate for sure, but we don't need to put it all in memory | |
| 21:53:22 | dansmith | sure | |
| 21:53:35 | dansmith | anyway, I'll look more at this when I get back | |
| 21:54:24 | bauzas | so, for example, to-be-weighting hosts could be like having an index and a weight so we could just sort based on that | |
| 21:54:39 | bauzas | instead of sorting based on a full memory | |
| 21:54:53 | bauzas | and once we have filtered hosts, we don't really need to keep that in memory | |
| 21:55:26 | bauzas | since for the moment the scheduler can hardly support more than 400 nodes, I think it's not a problem bubbling up | |
| 21:55:42 | bauzas | but if we say for cells v2 that all the nodes will be verified, then I think we should fix that | |
| 21:56:11 | bauzas | and please, don't coufound iterators and lists | |
| 21:57:14 | bauzas | edleafe: oh, and FWIW, the 2-tuple (host, alts) is just for a priv method | |
| 21:57:23 | bauzas | edleafe: select_dests is returning a keyed dict | |
| 21:57:32 | bauzas | well, a list of keyed dicts rather | |
| 22:03:50 | edleafe | bauzas: yes, I do believe I've seen that code once or twice :) | |
| 22:04:32 | edleafe | I was just asking if that implementation was agreed upon at the summit | |
| 22:05:18 | bauzas | we didn't discussed on the interface | |
| 22:05:30 | bauzas | so you could be bikeshedding it :p | |
| 22:06:12 | bauzas | tbc, I think that since _schedule() is private, we shouldn't really care of what we return | |
| 22:06:22 | edleafe | nope, just trying to catch up beyond the etherpad https://etherpad.openstack.org/p/BOS-forum-move-claims-from-compute-to-scheduler | |
| 22:06:25 | bauzas | the crux should be around select_dests() | |
| 22:46:57 | melwitt | anyone else interested in preventing KeyErrors in exceptions in unit test (creates garbled text to stdout), please review dis: https://review.openstack.org/#/c/411615 | |
| 23:00:41 | melwitt | woot, thanks | |
| 23:07:38 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Cache database and message queue connection objects https://review.openstack.org/465042 | |
| #openstack-nova - 2017-05-17 | |||
| 00:33:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing https://review.openstack.org/465205 | |
| 00:55:22 | armax | mriedem: ping | |
| 00:55:29 | mriedem | armax: pong | |
| 00:55:37 | armax | quick question | |
| 00:55:55 | armax | since I know how much you love numa and sriov code in nova | |
| 00:56:11 | armax | do you happen to know anything about this bug https://bugs.launchpad.net/mos/+bug/1658070? | |
| 00:56:11 | openstack | Launchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed] - Assigned to Sergey Nikitin (snikitin) | |
| 00:56:16 | armax | apparently it’s targeted to MOS | |
| 00:56:37 | armax | but I fear that’s plainly affecting nova | |
| 00:56:48 | mriedem | hmm, doesn't ring a bell, but i bet cfriesen might | |
| 00:57:13 | mriedem | i think snikitin is off openstack now too | |
| 00:57:24 | armax | ok thanks | |
| 00:57:45 | armax | I’ll poke around | |
| 00:58:02 | mriedem | doesn't mention which release either | |
| 00:58:41 | mriedem | mitaka i guess? | |
| 01:01:52 | mriedem | armax: hmm https://github.com/openstack/nova/commit/dfdae01bd03f522ffab7876b253ec41641934702 | |
| 01:01:58 | mriedem | ^ went back to newton | |
| 01:02:41 | armax | I see | |
| 01:02:48 | armax | that’s good to know | |
| 01:02:50 | mriedem | but that bug was around awhile | |
| 01:02:52 | mriedem | i remember | |
| 01:02:57 | mriedem | it was blocking migration with pci | |
| 01:03:07 | mriedem | https://bugs.launchpad.net/nova/+bug/1512880 | |
| 01:03:08 | openstack | Launchpad bug 1512880 in OpenStack Compute (nova) newton "Failed cold migration with SR-IOV" [Medium,Fix released] | |
| 01:03:34 | armax | that seems like it’s a slightly different use case | |
| 01:04:03 | mriedem | maybe | |
| 01:04:08 | mriedem | evacuate and migrate are pretty similar though | |
| 01:04:20 | armax | yeah | |
| 01:04:47 | mriedem | could also be some stuff in the resource tracker in the compute, but the bug report said that if they specify a host it fails, but if they don't it works | |
| 01:04:55 | armax | perhaps bug https://bugs.launchpad.net/mos/+bug/1658070 is newton? | |
| 01:04:56 | openstack | Launchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed] | |
| 01:04:57 | mriedem | moshele and cfriesen are definitely the guys to ask | |
| 01:05:07 | armax | as the workaround says to let nova figure out the host | |
| 01:05:15 | mriedem | yeah that part confuses me | |
| 01:05:45 | mriedem | unless when you don't specify a host, the scheduler picks the right how to migrate to | |
| 01:05:51 | mriedem | *host | |
| 01:06:31 | armax | right, though we’d have to figure out whether all hosts are equal or the failure is genuine | |
| 01:07:45 | armax | thanks for the feedback, I’ll try to get confirm whether this is an issue still affecting master | |
| 01:08:20 | mriedem | sure, if all else fails moshele in the morning (he's in israel i think) | |
| 01:08:31 | mriedem | cfriesen has no excuse for not being around right now working at all hours, he's in canada | |
| 01:11:14 | artom | Yeah, our country's so big we encompass all timezones, and therefore work around the clock | |
| 01:11:20 | mriedem | https://review.openstack.org/#/q/topic:bug/1512880+status:merged+branch:master | |
| 01:11:35 | mriedem | armax: ^ was really a whole series of changes to get cold migration working with pci devices | |
| 01:11:51 | mriedem | artom: amen | |
| 01:12:39 | armax | mriedem: so I guess we’ll have to see whether the issue exists on master or not | |
| 01:13:20 | mriedem | i wouldn't be surprised if it does, evacuate isn't integration tested | |
| 01:13:40 | mriedem | anything with pci/sriov is not tested heavily, it's all 3rd party and a limited scope i think | |
| 01:13:46 | armax | yup | |
| 01:13:51 | armax | makes sense | |
| 01:14:12 | mriedem | but the mellanox guys were running cold migration with sriov in their ci i think | |
| 01:14:22 | mriedem | but we can't test evacuate in tempest since you have to take down the single compute service | |
| 01:14:50 | armax | yeah, that’s not really reasonable on a continuous basis | |
| 01:14:54 | artom | Surely we can hack something around that... | |
| 01:15:03 | artom | But maybe it's just my late night brain talking | |
| 01:15:09 | mriedem | artom: yeah we've talked about ways to hack that | |
| 01:15:22 | mriedem | could be in the multinode job but you have to basically lock all other tests from running at the same time | |
| 01:15:37 | mriedem | would be best if we just had some integration tests to run serially in-tree | |