Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-08-24
14:34:53 dansmith dtantsur: the other thing that helps is that if you're really at 100% fail, eventually you stop wasting time, cpu, and network bandwidth trying to build and prepare things that are going to fail
14:34:57 dtantsur I'm reserving my opinion on this error :) it's too hard to debug, and it can mean anything (especially when RetryFilter is used)
14:35:18 dansmith so getting to zero disabled computes doesn't seem like a bad thing to me if literally 100% of the builds will fail
14:36:17 dtantsur does it include attempts from the RetryFilter?
14:36:51 dansmith I'm not sure what you mean.. are you asking if three retries against the same compute count as three failures?
14:37:05 dtantsur (side note: I became aware of this option after seeing https://review.openstack.org/#/c/496851/ )
14:37:11 dtantsur dansmith: yep
14:37:23 dansmith dtantsur: sure, it's all the same
14:38:07 dtantsur dansmith: meaning, if people have retry count >= 10, one misconfigured instance can bring down a compute node?
14:38:29 dansmith dtantsur: if they configure retries high and don't adjust this, then sure
14:39:13 dtantsur okay, I think this is at least worth mentioning in our docs, because it may cause surprises (well, it did cause surprises already for tripleo people)
14:39:46 openstackgerrit Ed Leafe proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810
14:39:59 dansmith dtantsur: what sort of mention do you want?
14:40:26 dansmith note that the text of that config option mentions retries,
14:40:34 dansmith so hopefully someone upping that to a crazy level will see that
14:41:08 dtantsur dansmith: I want to have a mention of it in https://docs.openstack.org/ironic/latest/install/configure-compute.html
14:41:32 dtantsur maybe just something like "Mind the following option, if you don't like <behavior>, change it to 0"
14:42:15 dansmith dtantsur: sure, ironic docs that talk about ideal nova settings when using the two together is certainly a good idea :)
14:42:43 dtantsur yep, that was my initial idea
14:42:52 dansmith I have no problem with that
14:43:50 alex_xu mriedem: would you mind take care of https://review.openstack.org/496995, since the time is late for me
14:44:34 dtantsur cool, thanks all!
14:45:20 mriedem alex_xu: yup, i'll take over, thanks for working on it
14:45:47 alex_xu mriedem: thanks, good luck for rc2 :)
14:45:49 mriedem mnestratov|2: fyi https://bugs.launchpad.net/nova/+bug/1712801
14:45:50 openstack Launchpad bug 1712801 in OpenStack Compute (nova) "Virtuozzo conainers lack SCSI info in libvirt XML" [Undecided,New]
14:46:36 dtantsur sahid: actually, could you take a quick look at https://docs.openstack.org/ironic/latest/install/configure-compute.html to check if everything is still correct there? there are a few options that I don't remember well
14:46:40 dtantsur oops, wrong ping
14:46:52 dtantsur sorry sahid, I meant dansmith (and please don't ask how I made this typo)
14:47:08 sahid :)
14:47:55 mnestratov|2 mriedem: thanks, this is our guy filed the bug in context of review https://review.openstack.org/#/c/495756/
14:49:06 dansmith dtantsur: we don't need the hostmanager thing after the resource class transition right?
14:50:21 dansmith I'm also not sure why you tell people to use a crazy host_subset_size without explaining why they may or may not want that
14:50:36 dansmith especially in pike where we shouldn't have any scheduler races
14:51:12 dtantsur I think using a value > 1 is still a good idea, as two instances cannot get on one host
14:51:28 dtantsur probably not so huge, and I agree that we need an explanation
14:51:52 dtantsur and yes, it seems like we should have deprecated ironic_host_manager in pike
14:52:53 lajoskatona cdent: thanks I check it
14:53:06 dansmith dtantsur: with claims in the scheduler and resource classes, you can not have two instances pick the same host
14:53:18 efried edleafe "Return a list of selected host + alternates, along with their allocations to the conductor" <== Is [allocations to the conductor] a thing, or are [a list of selected hosts + alternates, along with their allocations] being returned to the conductor?
14:53:43 efried (Trying to figure out if there should be a comma after 'allocations'.)
14:53:49 dansmith dtantsur: but without that, an insane host_subset_size just means there is zero weight applied to any host, which I would expect some people would not want
14:53:49 dtantsur dansmith: right, I may be severely outdated on this
14:54:20 edleafe efried: the latter
14:54:21 dtantsur so, what would you recommend, keeping it 1? or setting it to something moderate?
14:54:35 edleafe we return a whole big glob of stuff to the conductor
14:54:45 efried edleafe k, so yes comma. There's another typo; will post an edit.
14:55:09 dansmith dtantsur: I would rather you explain why you're prescribing one thing or the other, and maybe just call out that it's moot with RC usage
14:55:38 dtantsur dansmith: then I wonder why we had to do https://review.openstack.org/#/c/493989/.. maybe it was before we moved the CI to resource classes though..
14:55:41 openstackgerrit Eric Fried proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810
14:55:42 dtantsur vdrok: do you remember ^^^?
14:56:35 dtantsur or maybe it was because of backports...
14:56:37 dansmith dtantsur: yeah I dunno
14:56:49 openstackgerrit Eric Fried proposed openstack/nova master: Monkey patch the blockdiag extension https://review.openstack.org/476159
14:56:55 dtantsur I cannot combine my brain back in a working state after the release drill
14:58:08 dansmith dtantsur: for devstack there's likely no reason to weigh any of the options and so increasing that value just means we make the scheduler choose randomly amongst empty hosts
14:58:15 dansmith which might be better for the gate for some reason, I dunno
14:58:33 dansmith but not for regular people that might want to weigh hosts with newer hardware differently or something like that
14:59:04 dtantsur weighing hosts for ironic is something new to me, but it makes sense indeed
14:59:33 dansmith dtantsur: well, maybe it's new because you disable weighing entirely with that 999999 thing :P
14:59:34 edleafe efried: I can't vim
14:59:44 efried edleafe :)
15:00:02 dtantsur dansmith: haha, no, I did not even know it was possible. have I ever mentioned my poor knowledge of nova?
15:00:12 dansmith dtantsur: I don't know that our in-tree weighers would mean anything for ironic nodes, but weighers are some of the more likely things for someone in a deployment to write customly to get the host selection they want
15:00:18 dansmith heh
15:00:42 dtantsur yeah, and "prefer newer hardware" makes some sense to me indeed, for example
15:00:58 dansmith yep
15:01:19 openstackgerrit sahid proposed openstack/nova master: libvirt: add method to configure migration speed https://review.openstack.org/497456
15:01:19 openstackgerrit sahid proposed openstack/nova master: libvirt: bandwidth param should be set in guest migrate https://review.openstack.org/497455
15:01:20 openstackgerrit sahid proposed openstack/nova master: libvirt: slowly live-migration to ensure network is ready https://review.openstack.org/497457
15:01:27 dtantsur okay, thanks again dansmith. may I ping you for review when I write an update for this docs page?
15:01:38 dansmith dtantsur: sure
15:01:50 dtantsur cool, I'll try to finish it today
15:02:00 openstackgerrit David Rabel proposed openstack/nova master: Adds support for graceful shutdown for VMware instances https://review.openstack.org/494169
15:06:21 mriedem here is the bug for the resize + reschedule not removing allocations issue https://bugs.launchpad.net/nova/+bug/1712850
15:06:22 openstack Launchpad bug 1712850 in OpenStack Compute (nova) "Allocations are not removed from destination node when rescheduling during resize/migrate" [High,Triaged]
15:07:08 vdrok dtantsur: /me is on holiday today, but it was needed because all weights are the same in ci for our nodes, so the same ones were selected during parallel tests for instance build, and were racing as iirc claim happens on compute
15:07:51 vdrok So reschedules were happening constantly, and Max reschedules is 3 by default
15:08:04 rabel is "Intel PCI CI" always non-voting?
15:09:13 rabel that is: can i ignore that it failed without a reason? according to https://wiki.openstack.org/wiki/ThirdPartySystems/Intel-PCI-CI it is non-voting
15:09:33 cdent rabel: pretty much, yeah
15:09:49 rabel cool, thanks.
15:10:18 mriedem rabel: you can yell at sean-k-mooney about that
15:14:27 dtantsur dansmith: see vdrok's comment above ^^^. is it going away with resource classes? especially wrt "claim happens on compute"?
15:14:41 sean-k-mooney haha well its not run by our team but what was the questing releating to the pci ci
15:15:17 rabel the question was just if i can ignore if it fails. ;)
15:15:17 dansmith dtantsur: yep
15:15:33 dtantsur good to know
15:15:54 sean-k-mooney ah well yes and no. if you are not touching the pci code probably if you are you might want to take a look at it
15:16:12 sean-k-mooney the pci ci is testing a very limited scope
15:18:17 openstackgerrit Eric Fried proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810
15:18:19 sean-k-mooney we had planned to combine it with the nfv ci and expand the scope to introduce livemigration and other testing but we lost 2 headcount including our ci lead so that did not go ahead
15:18:59 dtantsur dansmith: why aren't compute_group_opts here https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L1122-L1130 ? I think it causes this option to not appear in the sample conf
15:19:39 dansmith dtantsur: I dunno what sort of black magic is involved in all this.. maybe ask mriedem or sfinucan or edleafe
15:20:07 dtantsur ack
15:22:14 dtantsur I also see compute_driver option twice - in [DEFAULT] and in [compute], wut?
15:23:50 mriedem yeah that looks like a bug, the former i mean
15:24:51 dtantsur mriedem: well, actually the option is in [DEFAULT]. lemme bake a quick patch, you'll see what I mean
15:25:17 efried mriedem dtantsur L1141 should be compute_group_opts, not compute_opts.
15:25:32 dtantsur this is what I'm fixing right now, yes :)

Earlier   Later