| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-24 | |||
| 13:07:44 | sean-k-mooney | ykarel: you realsie that we are not ment to have any voting job that use nested virt since we have not enough ci priorse for it. we were asked in teh past not to make any voting jobs unless we had 3 providers for it | |
| 13:08:05 | sean-k-mooney | it looks like we now have 2 clouds form vexhost and one form ovh | |
| 13:08:15 | sean-k-mooney | is this why this was made to use nested virt | |
| 13:08:22 | ykarel | sean-k-mooney, when we moved i recall there were 3 providers, need to check current status | |
| 13:08:26 | sean-k-mooney | i am not aware fo any anochment that it was now ok to do this | |
| 13:08:50 | sean-k-mooney | ykarel: so nested virt should only be used if a job needs it | |
| 13:09:02 | sean-k-mooney | what in this tempest plug need nested virt | |
| 13:09:59 | sean-k-mooney | we were previosuly asked not to enable it just to make the job faster to concerve the capsity | |
| 13:10:09 | ralonsoh | no one in particular, that was to improve the current stability of our jobs | |
| 13:10:09 | ykarel | sean-k-mooney, yes right nothing in those tests need nested virt, the switch was done just for better perf/times in jobs, and that worked great for us for almost an year before hte switch to jammy | |
| 13:10:29 | sean-k-mooney | ykarel: right so per/jobs times was specfic not a valid reason to use it in the past | |
| 13:11:00 | ralonsoh | ok, let's revert that for now and move to jammy | |
| 13:11:15 | ralonsoh | I'll push a DNM patch to test that | |
| 13:11:42 | lajoskatona | +1 let's have fresh results, and base any forward steps on that | |
| 13:14:23 | ykarel | sean-k-mooney, yes right i discussed this few months back with infra some months back and i was said it should be avoided just for perf but if team knows the issues with using nested virt(no support, less providers etc) it can be used as those nodes have resources available to use | |
| 13:15:08 | sean-k-mooney | ykarel: it used ot be stated in the docs somewhere too | |
| 13:15:53 | ykarel | and at that time we switched to focal until infra issue get's fixed(it requires infra compute nodes to be upgraded) | |
| 13:15:54 | sean-k-mooney | ykarel: my concern is we have thing that at least previous could only be tested with nested virt in nova | |
| 13:16:09 | sean-k-mooney | and it was previously not considerd ok to use the nested virt lables to do that | |
| 13:17:55 | ykarel | ralonsoh, lajoskatona due to qemu/libvirt version just revert will not work as we run many concurrent guest vms (6+) while qemu-5.0.0 it uses too much memory | |
| 13:18:03 | ykarel | 1gb per guest vm extra by default | |
| 13:18:29 | sean-k-mooney | because fo the tcib cache issue | |
| 13:18:33 | sean-k-mooney | when not using nested | |
| 13:18:52 | ykarel | libvirt-8.0.0 supports customizing it and for that DNM patch https://review.opendev.org/c/openstack/nova/+/868419 | |
| 13:19:10 | ykarel | will update it as per sean-k-mooney comments | |
| 13:19:35 | sean-k-mooney | this should be a specless blueprint | |
| 13:19:38 | ykarel | but even with we need to adjust(split into multiple ) our jobs | |
| 13:19:47 | sean-k-mooney | can we get this on the team meetign adjenda for tomorrow | |
| 13:19:51 | sean-k-mooney | bauzas: ^ | |
| 13:20:29 | sean-k-mooney | ykarel: i think we shoudl start a mail thread on using nested virt in votign jobs | |
| 13:22:01 | sean-k-mooney | https://lists.openstack.org/pipermail/openstack-discuss/2019-April/004681.html was the last time we discussed it on the list i think | |
| 13:26:25 | bauzas | sean-k-mooney: I'm a bit diverted by some meeting now, but agenda is up for everyone : https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 13:27:30 | sean-k-mooney | bauzas: so tooz shoudl be blocked by UC right | |
| 13:28:06 | sean-k-mooney | i guess we dont have any focal based job that blocked the update | |
| 13:28:39 | sean-k-mooney | so in the short term i think we are goign to have to downgrwade the tooz verion in uc untill we can move the ceph jobs to focal or resolve the uwsgi issue | |
| 14:21:55 | dansmith | bauzas: any ideas on who else we could get to review this and the following patch? https://review.opendev.org/c/openstack/nova/+/880632 | |
| 14:22:04 | dansmith | I'm sort of waiting to rebase my actual compute ids stuff on that | |
| 14:22:28 | bauzas | dansmith: good question, let's say gibi or sean-k-mooney | |
| 14:22:39 | ykarel | sean-k-mooney, ack will add it to meeting agenda. and wrt mail thread sure could be done but as discussed in past voting should be avoided unless and until nested-virt is really needed as it's support is best effort and only few providers compared to other providers, but may be it can help in getting the messaging more clear to everyone | |
| 14:22:52 | dansmith | bauzas: ack, I just know they're both very busy with other things.. I added sean-k-mooney a while back | |
| 14:45:54 | frickler | sean-k-mooney: there could also be a specific py38 pin in u-c for tooz, like we had for py2.7 for some time. you may want discuss with reqs team (or maybe TC) | |
| 14:47:51 | frickler | also reminder that tooz isn't the only lib affected, oslo.db has the same situation except that u-c adoption is still blocked by other things stephenfin is having fun with | |
| 15:07:25 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | |
| 15:23:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Revert "Temporary skip some volume detach test in nova-lvm job" https://review.opendev.org/c/openstack/nova/+/881389 | |
| 15:43:44 | opendevreview | Balazs Gibizer proposed openstack/nova master: Revert "Temporary skip some volume detach test in nova-lvm job" https://review.opendev.org/c/openstack/nova/+/881389 | |
| 16:07:29 | bauzas | sean-k-mooney: dansmith: lemme get it right, so https://zuul.opendev.org/t/openstack/build/5f269ed5244d47be800dab988253c72c is failing because we bumped py to 3.9 and apache httpd complains ? https://review.opendev.org/c/openstack/nova/+/881339/3/.zuul.yaml#586 | |
| 16:07:52 | sean-k-mooney | yes | |
| 16:08:04 | sean-k-mooney | if we move tooz to an extra package | |
| 16:08:10 | sean-k-mooney | we can drop the bump to 3.9 | |
| 16:08:21 | sean-k-mooney | and that job should work since it does not have ironic | |
| 16:08:27 | sean-k-mooney | or barbican | |
| 16:08:38 | sean-k-mooney | so castalan and tooz wont be installed | |
| 16:09:10 | sean-k-mooney | or we can move the nova-ceph-multistore to 22.04 | |
| 16:09:20 | dansmith | I don't actually see the failure logged anywhere, other than keystone can't be imported | |
| 16:09:55 | bauzas | yeah me too, hence my question | |
| 16:10:05 | dansmith | oh I see that's the bump to 3.9 | |
| 16:10:09 | bauzas | this looks to me that keystone can't work with 3.9 | |
| 16:10:22 | dansmith | or uwsgi is using 3.8 more likely I think | |
| 16:10:30 | bauzas | because RegionOne isn't an UUID | |
| 16:10:38 | dansmith | it says importerror but doesn't actually show that it's not a valid module, so I'm not positive | |
| 16:10:51 | dansmith | either way, I feel like 3.9 on focal isn't really an option | |
| 16:10:53 | sean-k-mooney | dansmith: thats failing because uwsgi si installed under python 3.8.10 | |
| 16:10:58 | sean-k-mooney | Python version: 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0] | |
| 16:11:10 | dansmith | sean-k-mooney: yup | |
| 16:11:22 | sean-k-mooney | whihc i think you said was a compile time dep | |
| 16:11:26 | sean-k-mooney | for uwsgi | |
| 16:11:30 | dansmith | no, I didn't say that | |
| 16:11:31 | bauzas | fwiw, the u-c bump to tooz==4.0.0 is still on hold | |
| 16:11:44 | bauzas | so we may ask for a cap | |
| 16:11:45 | dansmith | I said some of our packages we install from bindep to avoid compiling things like mysqlclient (AFAIK) | |
| 16:11:54 | sean-k-mooney | ah | |
| 16:12:06 | sean-k-mooney | ya sorry you did mention mysql | |
| 16:12:11 | sean-k-mooney | not uwsgi | |
| 16:12:16 | dansmith | I think the uwsgi python module *is* tied directly to the python version though | |
| 16:12:27 | dansmith | so we'd need a python3.9-uwsgi-module-python3 (or whatever the name is) | |
| 16:12:32 | sean-k-mooney | at least in the disto package i think that is correct | |
| 16:13:16 | dansmith | uwsgi-plugin-python3 | |
| 16:13:28 | dansmith | this ^ is compiled directly against the base python3, not 3.9 | |
| 16:14:46 | dansmith | see the last answer here: https://stackoverflow.com/questions/68413988/building-a-uwsgi-plugin-for-python-3-9-failed-for-older-version-it-works-are-th | |
| 16:14:55 | dansmith | might be able to pip install uwsgi with the 3.9 python | |
| 16:14:55 | sean-k-mooney | for the devstack venv path i think you can override that with https://review.opendev.org/c/openstack/devstack/+/558930/26/files/apache-horizon.template | |
| 16:15:08 | sean-k-mooney | WSGIPYTHONHOME | |
| 16:15:14 | dansmith | I' | |
| 16:15:17 | dansmith | I am pretty sure not | |
| 16:15:30 | dansmith | uwsgi runs the python interpreter internally, AFAIK | |
| 16:15:44 | dansmith | that's how it provides its phantom importable modules, etc | |
| 16:15:50 | sean-k-mooney | i know i was doing somethign to get it too work in that seriess | |
| 16:15:55 | sean-k-mooney | but i dotn recall | |
| 16:16:31 | sean-k-mooney | oh it was this https://review.opendev.org/c/openstack/devstack/+/558930/26/functions-common#1605 | |
| 16:16:38 | sean-k-mooney | anywya that wont really help here | |
| 16:16:54 | sean-k-mooney | for the ceph job we whould rever back to 3.8 i guess | |
| 16:17:05 | sean-k-mooney | if we are keeping it on focal | |
| 16:18:28 | sean-k-mooney | mixing the workaround to run with uwsgin for a venv and the ceph job is not helpful i tought there was a simple fix in that but no | |
| 16:19:17 | bauzas | stupid question but I guess we can't block a specific tooz version in our own requirements.txt file ? | |
| 16:19:45 | sean-k-mooney | we can but we are not ment too | |
| 16:19:53 | sean-k-mooney | we can do tooz!=whatever | |
| 16:20:08 | sean-k-mooney | that will cause use to downgrade if its already installed | |
| 16:20:12 | bauzas | and tooz<=version ? | |
| 16:20:22 | sean-k-mooney | we dont want to cap | |
| 16:20:24 | dansmith | right but we'll still install the initial version first, then downgrade it, | |