| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-08 | |||
| 14:43:16 | cdent | mriedem: I'm continuing my explorations with slamming placement etc. and am now wondering: What are the reasons (if any) that we don't allow nova-scheduler to have worker processes? | |
| 14:43:52 | bauzas | cdent: IIRC, that was because we were having racinesses | |
| 14:44:02 | bauzas | I can find old threads | |
| 14:44:10 | cdent | bauzas: thanks | |
| 14:44:26 | mriedem | here you go https://review.openstack.org/#/c/159382/ | |
| 14:44:29 | mriedem | cdent: bauzas: ^ | |
| 14:44:40 | mriedem | that's a huawei patch so i've got'er starred :) | |
| 14:44:44 | bauzas | ideally, I just feel we should pursue the idea to drop the whole n-sch service and do the thing in the conductor | |
| 14:45:04 | stephenfin | dansmith: Highlighted the RPC 5.0-ish things I'm aware of here https://review.openstack.org/#/c/541005/ | |
| 14:45:13 | mriedem | cdent: fwiw, i've had a todo sitting here to update nova-next in devstack to run with 2 nova-scheduler processes | |
| 14:45:21 | dansmith | stephenfin: thanks will look in a sec | |
| 14:45:33 | cdent | forcing some workers makes the rpc timeout problems go away and the instances to land, but I've got all filters turned off, so I suspect raciness won't rear its head if it is still there | |
| 14:45:37 | bauzas | cdent: in particular if you split placement, having multiple n-sch besides conductors is a PITA for operators | |
| 14:45:41 | stephenfin | Not sure if that can be done or not, so I'm relying on you to educate me. Can tackle it myself though, if so (and if it would help) | |
| 14:46:17 | bauzas | cdent: mriedem: the problem was tied to the HostManager.consume_from_instance() lock, IIRC | |
| 14:46:19 | cdent | bauzas: you say that but I find it hard to believe in a world of automation that running another service is a big deal, especially in the face of so many services... | |
| 14:46:33 | bauzas | call it microservices :) | |
| 14:46:52 | openstackgerrit | Jianghua Wang proposed openstack/nova master: VGPU: Modify the example of vgpu white_list set https://review.openstack.org/539183 | |
| 14:47:06 | mriedem | most places are running multiple schedulers, they have to for the load, they just get reschedules if they are packing | |
| 14:47:19 | jianghuaw | bauzas, mriedem: ^ | |
| 14:47:21 | ameeda | mriedem: when I try to do "tox bindep" I got those errors http://paste.openstack.org/show/666615/ , I use master branch "queens" , any idea ? | |
| 14:47:29 | bauzas | multiple schedulers != multiple workers, and I don't exactly remember why | |
| 14:47:51 | bauzas | that's old plain story I don't remind | |
| 14:48:57 | stephenfin | ameeda: What host OS are you using? | |
| 14:49:12 | ameeda | Centos , should I do it on ubuntu ? | |
| 14:49:24 | cdent | mriedem, bauzas : do you recall if Nikola ever produced the email message or blog post that was claimed? | |
| 14:49:26 | bauzas | mriedem: oh thanks for the patch, yeah I got the problem now | |
| 14:50:07 | stephenfin | Not yet. Try adding 'basepython = python2.7' above 'deps' in the '[testenv:bindep]' section of tox.ini | |
| 14:50:10 | bauzas | cdent: what do you mean by "claimed" ? | |
| 14:50:12 | stephenfin | ameeda: ^ | |
| 14:50:25 | bauzas | cdent: I can point you some ndipanov's changes about fixing the locking mechanism | |
| 14:50:32 | stephenfin | ameeda: That'll force tox to use Python 2.7 | |
| 14:50:35 | bauzas | that's related | |
| 14:50:39 | cdent | bauzas: he said in the comments that he would right a more detail comment in an email message. did he? | |
| 14:50:49 | stephenfin | Actually, nvm me | |
| 14:50:57 | stephenfin | ameeda: It's 'tox -e bindep' | |
| 14:50:58 | bauzas | cdent: don't blame me, I'm 37 but I have poor memory :) | |
| 14:51:24 | ameeda | stephenfin: thanks !! | |
| 14:51:49 | bauzas | but I know we merged some concurrency patches around consume_from_instance, which later became consume_from_request() | |
| 14:52:40 | openstackgerrit | Jianghua Wang proposed openstack/nova master: VGPU: Modify the example of vgpu white_list set https://review.openstack.org/539183 | |
| 14:52:48 | dansmith | stephenfin: the instance object stuff is definitely clear, and I think host as well, but if you could track down the host bit that would be appreciated | |
| 14:53:02 | stephenfin | dansmith: Sure thing | |
| 14:53:13 | ameeda | stephenfin: Missing packages: postgresql-server python34-devel sqlite-devel ERROR: InvocationError: '/opt/stack/nova/.tox/bindep/bin/bindep test' | |
| 14:53:21 | dansmith | even so, I'm not sure those compat blocks are fully legit, as we've had obj_compat decorators to hide that stuff for many year | |
| 14:53:32 | cdent | Well, for now I will continue my experiments with multiple scheduler workers and see what happens. | |
| 14:54:12 | stephenfin | ameeda: Right, so bindep isn't tracking the correct packages for CentOS, That's another bug | |
| 14:54:39 | stephenfin | ameeda: But given the issue was reported against Ubuntu, you probably _should_ use that to fix _that_ particular bug :) | |
| 14:54:46 | bauzas | cdent: I think I found the ML thread | |
| 14:54:50 | ameeda | stephenfin: I will test it on ubuntu | |
| 14:55:10 | ameeda | stephenfin: do you know the link of current centos bug :$ | |
| 14:55:11 | jianghuaw | bauzas, have removed the quotes:-) https://review.openstack.org/#/c/539183 | |
| 14:55:19 | cdent | bauzas: awesome | |
| 14:55:23 | bauzas | cdent: http://thread.gmane.org/gmane.comp.cloud.openstack.devel/47882 | |
| 14:55:30 | stephenfin | ameeda: Nope, there probably isn't one. You should report it (and maybe try to fix it) | |
| 14:55:52 | ameeda | stephenfin: great, Thanks for help :) | |
| 14:55:53 | stephenfin | Assuming we support CentOS for that. I'm not sure if we do, given that we don't gate on it | |
| 14:55:56 | stephenfin | np | |
| 14:56:09 | mriedem | just add the rpm and deb packages to bindep in the same patch | |
| 14:57:16 | mriedem | libpcre3-dev [platform:dpkg] | |
| 14:57:19 | mriedem | i think that's all you need | |
| 14:58:03 | mriedem | looks like it's pcre-devel on fedora/centos | |
| 14:58:06 | mriedem | https://rpmfind.net/linux/rpm2html/search.php?query=pkgconfig(libpcre) | |
| 14:58:13 | ameeda | mriedem: you are talking about current opened bug of ubuntu . what do you think about the Centos one ? | |
| 14:58:30 | mriedem | ameeda: it's a nova bug that the entries are missing from nova's bindep, | |
| 14:58:38 | mriedem | just add the entries for deb and rpm | |
| 14:58:43 | mriedem | which i've given above | |
| 14:59:01 | ameeda | to this file bindep.txt right ? | |
| 14:59:07 | mriedem | yes | |
| 14:59:14 | mriedem | see the existing entries in there as an example | |
| 14:59:17 | ameeda | cool, thanks | |
| 14:59:59 | mriedem | same for opensuse https://software.opensuse.org/package/pcre-devel | |
| 15:00:21 | mriedem | looks like we don't have specific entries for suse | |
| 15:00:53 | sean-k-mooney | stephenfin: since the intel nfv ci has been offline we dont test int 300 times a day and i have only been using it on ubuntu lately but i fixed the centos support a few months ago when centos decided qemu-kvm should nolonger install qemu.. | |
| 15:01:01 | sean-k-mooney | stephenfin: if you have issue let me know | |
| 15:01:08 | Roamer` | mriedem, pcre-devel should cover all RPM-based distributions | |
| 15:01:09 | mriedem | ameeda: btw, you can see the same thing done in devstack https://github.com/openstack-dev/devstack/commits/master/files | |
| 15:01:22 | stephenfin | sean-k-mooney: I'm using CentOS, so I sure will | |
| 15:01:31 | mriedem | devstack changes aren't bindep, but that shows you which packages for which distro | |
| 15:01:39 | Roamer` | mriedem, there's a bit less divergence in RPM-based distributions than there is between Debian and Ubuntu sometimes, at least WRT libraries and development packages, not the system startup stuff :) | |
| 15:02:29 | ameeda | mriedem: now I am using Centos, and the bug for Ubuntu , so the fix needs to be on ubuntu or don't care ? | |
| 15:02:35 | Roamer` | (and I say this a bit sadly, with all my love for Debian and respect for Ubuntu and, well, reluctance when it comes to working on RPM-based distributions) | |
| 15:02:57 | mriedem | ameeda: i'm not sure how to answer that | |
| 15:03:11 | mriedem | ameeda: you can update nova and push the fix from a gentoo system, it doesn't matter, | |
| 15:03:18 | mriedem | just add the packages to nova's bindep file | |
| 15:03:22 | Roamer` | ameeda, try to figure out what the bug actually is, what libpcre3-dev is and why it needs to be added | |
| 15:03:36 | mriedem | Roamer`: it's a transitive dependency, | |
| 15:03:42 | Roamer` | mriedem, I know, I fixed it for our CI | |
| 15:03:43 | mriedem | because of the new dep on the whereto package | |
| 15:03:51 | mriedem | whereto requires pcre | |
| 15:03:59 | cdent | bauzas: thanks for that. I read through and I wonder to what extent claims in the scheduler are now making a difference to the concerns? | |
| 15:04:02 | ameeda | thanks all for help, I have to go now, I will back soon. | |
| 15:06:11 | bauzas | cdent: I feel we solved the main blocker | |
| 15:06:13 | gibi | mriedem: I think it is not a big deal but we have some inconsistency here https://review.openstack.org/#/c/541008/9/nova/tests/unit/image/test_glance.py@1629 | |
| 15:06:22 | bauzas | cdent: because I was wrong in the past | |
| 15:06:35 | bauzas | cdent: the fact that we have workers doesn't mean greenlets | |
| 15:06:53 | bauzas | hence separate processes, exactly like having multiple n-sch services | |
| 15:07:21 | bauzas | so, what we solved with scheduler claims is also a valid argument for saying we can have workers | |
| 15:10:57 | cdent | I've made reference to the code and the email thread in the notes I'm writing up, so we'll have that to refer back to later if needed | |
| 15:11:12 | mriedem | gibi: i think it's probably fine, the only place we append anything to that url is in "generate_image_url" and we append /images/{id} to it | |
| 15:11:23 | mriedem | for the createImage response location header | |