| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-10-24 | |||
| 18:36:58 | sean-k-mooney | the nova unit/funtional test failure might eb related or flaky tests so i woudl try at least one recheck to confirm that | |
| 18:39:31 | sean-k-mooney | frickler: the fact the devstack functional job worked is also odd | |
| 18:40:03 | dansmith | JayF: sean-k-mooney: https://github.com/openstack/nova/blob/icehouse-eol/nova/cmd/__init__.py | |
| 18:40:05 | dansmith | os=False | |
| 18:40:39 | sean-k-mooney | rigth so we still do that for winwos | |
| 18:40:42 | sean-k-mooney | *windows | |
| 18:40:58 | dansmith | but that was for every binary on every platform, AFAIK | |
| 18:41:00 | JayF | I think that was required to be added some point around t? u? v? era-ish for greendns to behave properly? My memory is very lossy but I remember having to back os=False outta IPA at some point | |
| 18:41:03 | dansmith | ...at the time | |
| 18:41:20 | sean-k-mooney | it was yes | |
| 18:41:28 | sean-k-mooney | im trying to see when we changed it | |
| 18:41:43 | sean-k-mooney | we sould be able to go back but i am not sure its breaking nova | |
| 18:42:22 | dansmith | it was that way through rocky at least | |
| 18:42:42 | dansmith | I lose the thread when the refactor to monkey_patch.py happened | |
| 18:42:48 | dansmith | anyway, | |
| 18:43:05 | dansmith | I'm not saying we should, I'm defending why I thought the os.open thing wouldn't affect us :) | |
| 18:43:08 | dansmith | and proving that I'm old | |
| 18:43:22 | sean-k-mooney | dansmith: the sdk devstack tests passed https://5355e5989bc73779bf10-cc682349d9014dd293929a9a1be5247d.ssl.cf5.rackcdn.com/933257/1/check/openstacksdk-functional-devstack/fad2996/testr_results.html | |
| 18:46:20 | sean-k-mooney | dansmith: https://github.com/openstack/nova/commit/4f2d4d03f3a1a0a277dd04ec2abcbee2d4018fa2 | |
| 18:47:27 | sean-k-mooney | so there was no reason given why it orginally was false and also no reason given why we shoudl remov it | |
| 18:47:38 | dansmith | yeah :) | |
| 18:47:55 | sean-k-mooney | dansmith: by the way i also remember os=false for what its worth | |
| 18:48:24 | sean-k-mooney | as in that what i expected to see on master | |
| 18:48:52 | sean-k-mooney | for now i just recheck the requiremnt patch to see if things are failing the same way consitently | |
| 18:49:51 | sean-k-mooney | hum we changed in stein | |
| 18:58:26 | opendevreview | Ghanshyam proposed openstack/osc-placement master: Update python classifier in setup.cfg https://review.opendev.org/c/openstack/osc-placement/+/931113 | |
| 19:12:03 | sean-k-mooney | frickler: i was able to repoduce the nova.tests.unit.test_service.TestWSGIServer.test_server_pool_waitall failure locally | |
| 19:13:30 | sean-k-mooney | nova.tests.unit.test_service.TestWSGIService has two failing unit tests for me too because we are not stubing calsl to bind but that happened with our without the new eventlet | |
| 19:24:15 | frickler | sean-k-mooney: maybe the results also vary between python versions/distros. the cross jobs run with py311 on bookworm | |
| 19:28:33 | sean-k-mooney | im using debian testing on 3.12 | |
| 19:28:46 | sean-k-mooney | frickler: the unit test fialure is because its too specific | |
| 19:29:03 | sean-k-mooney | the unit test is ment to assert that waitall is called when we shutdown the wsgi server | |
| 19:29:17 | sean-k-mooney | but its asserting its called once and now its called twice | |
| 19:29:25 | sean-k-mooney | so the fix for that is just change the assert | |
| 19:29:39 | sean-k-mooney | im repoducing the functional test failure now | |
| 19:40:38 | sean-k-mooney | the functionatl test that is failing is restarting the scheduler and that reulst in self.service_ref being None | |
| 19:40:55 | sean-k-mooney | i think this is an artifact of how we are doing that | |
| 19:41:45 | sean-k-mooney | restarting services in our functional tests is kind of a hack | |
| 19:42:19 | sean-k-mooney | so again i think this will be a case of a slight test tweak but not an actual regression | |
| 19:47:37 | opendevreview | sean mooney proposed openstack/nova master: [eventlet] update nova tests for eventlet 0.37.0 https://review.opendev.org/c/openstack/nova/+/933365 | |
| 19:47:45 | sean-k-mooney | frickler: ^ i think that will do it | |
| 19:49:13 | sean-k-mooney | i dont think that matrially changes the intent of any of those tests so if that passes ci we can merge it and at least get those jobs passign with eventlet 0.37.0 | |
| 19:49:25 | sean-k-mooney | i can try installing that in a devstack tomorrow or next week | |
| 19:49:43 | sean-k-mooney | assumign the requirements patch does not pass tempest-py3 this time | |
| 19:52:31 | sean-k-mooney | ok im going to call it a day chat to folks tomorrow o/ | |
| 20:47:06 | gouthamr | bauzas: delayed response wrt your question regarding the virtiofs topic.. there's a conflict tomorrow at 1400 UTC tomorrow, and there are TC sessions beyond that.. I don't mind us chatting about this async / over email.. I mainly wanted to understand where it falls in the team's priorities and what we/anyone can do to help | |
| #openstack-nova - 2024-10-25 | |||
| 02:37:35 | opendevreview | Ghanshyam proposed openstack/nova master: Update gate jobs as per the 2025.1 cycle testing runtime https://review.opendev.org/c/openstack/nova/+/932648 | |
| 03:07:00 | opendevreview | Ghanshyam proposed openstack/nova master: Update gate jobs as per the 2025.1 cycle testing runtime https://review.opendev.org/c/openstack/nova/+/932648 | |
| 03:26:26 | opendevreview | Takashi Kajinami proposed openstack/nova master: Stop asking bug report to upstream https://review.opendev.org/c/openstack/nova/+/933381 | |
| 03:26:42 | tkajinam | ^^^ I wonder if can consider this switch now | |
| 05:04:59 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Use native DictOpt for disk_cachemode https://review.opendev.org/c/openstack/nova/+/915390 | |
| 05:05:05 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Validate valid cache mode at config input layer https://review.opendev.org/c/openstack/nova/+/933327 | |
| 06:16:21 | bauzas | tkajinam: gibi: fwiw, the i18n team will have a session at 7am UTC related to the documentation translation, I'll be there | |
| 06:18:58 | bauzas | tkajinam: sure about the launchpad bug, but I have some concern | |
| 06:25:46 | tkajinam | bauzas, thx. I need to attend an internal meeting soon but if that finishes early then I'll join it | |
| 06:26:35 | tkajinam | we used to have number of people here in Japan who lead the translation work but most of these already left. that's a sad part of the discussion :-( | |
| 06:27:09 | tkajinam | bauzas, yeah there would be a tradeoffs but as we see more case where a reporter says " I was asked to report a bug so I did it" I think setting a better expectation might be helpful for both sides | |
| 06:31:24 | bauzas | tkajinam: yup indeed a tradeoff, I'll try to review your patch this morning my time | |
| 06:56:01 | bauzas | I wonder how many I'll have | |
| 07:00:22 | bauzas | people can chime in if they want | |
| 08:13:26 | gibi | bauzas: sorry but that was a late notice for me | |
| 08:13:36 | bauzas | gibi: no worries at all | |
| 08:14:13 | bauzas | given the weekdays we have, that's abolutely normal to take some break | |
| 08:14:46 | bauzas | one day we will have physical PTGs and that won't longer be a problem (half kidding) | |
| 08:16:18 | bauzas | anyway, the most important bit for nova to know is that translation patches to the doc would have a specific gerrit policy where language moderators would have a gerrit label | |
| 08:16:41 | bauzas | so a single core approval could be done based on the label | |
| 08:17:04 | bauzas | nothing harder than the current model we have with .pot files | |
| 08:19:05 | gibi | ack, thanks for the summary | |
| 08:53:08 | r-taketn | Hi, I want to use MIG(multi-instance GPU) on OpenStack VM by using NVIDIA A100. Must I install NVIDIA vGPU driver to use MIG on OpenStack VM? I cannot see /sys/class/mdev_bus, so I cannot use vGPU method . And I cannot see mig device by lspci so I cannot use pci-passthrough. | |
| 09:05:57 | pas-ha[m] | r-taketn: you must install latest NVIDIA AI Enterprise drivers on the compute node, and then switch the card to the MIG mode. Then partition it to MIGs. | |
| 09:07:09 | pas-ha[m] | the vGPU drivers no longer support A100 (for about a year already), everything that is MIG-capable requires installation of NVIDIA AI Enterprise | |
| 09:07:49 | pas-ha[m] | https://docs.nvidia.com/ai-enterprise/latest/user-guide/index.html | |
| 09:36:19 | r-taketn | pas-ha[m]: Thank you for reply. I read AI Enterprise driver manual. I only installed free nvidia-driver, but it looks like I could partition a GPU device to MIGs by using `nvidia-smi -i 1 -mig 1` and `nvidia-smi mig -i 0 -cgi 2g.20gb -C`. Is it just hardware-level partitioning? So must I install not-free AI Enterprise driver to use MIG on VMs or applications? | |
| 09:39:42 | pas-ha[m] | honestly not sure how would that work with 'free' driver. did you actually tried to perform those commands? the command may be there (nvidia-smi is installed even by the desktop GPU drivers), but does it do anything? anyway, you must switch the card to the SR-IOV mode first for it to work in the MIG mode `/usr/lib/nvidia/sriov-manage -e <PCI-ADDRESS-OF-NVIDIA-CARD>` | |
| 09:41:15 | pas-ha[m] | then you will get 16 extra PCI devices for this card, those are VFs for SR-IOV, you will create the MIGs on them and pass these VF PCI devices to the VMs. | |
| 09:52:10 | bauzas | r-taketn: sorrry was in a meeting | |
| 09:52:28 | bauzas | MIG-mode is somehow doable with Nova | |
| 09:52:42 | bauzas | but you need to pre-create the GIs | |
| 09:53:07 | bauzas | as pas-ha[m] said, you also need to pre-create the VFs | |
| 09:53:41 | bauzas | once you create the graphical instances, you should be able to use the related mdevs | |
| 09:55:22 | bauzas | please read https://docs.nvidia.com/ai-enterprise/5.1/user-guide/index.html#configuring-vgpu-manager-linux-with-kvm for how to enable VFs and GIs | |
| 09:56:08 | bauzas | for MIG, this is a bit later in the nvidia doc : https://docs.nvidia.com/ai-enterprise/5.1/user-guide/index.html#configuring-a-gpu-for-mig-backed-vgpus | |
| 10:00:52 | r-taketn | pas-ha[m], bauzas: I already created GIs. Sorry, I didn't understand mdev types and VFs. So I try to enable SR-IOV and create the VFs. Thanks. | |
| 10:32:29 | r-taketn | bauzas, pas-ha: I cannot find /usr/lib/nvidia/sriov-manage on my environment. https://docs.nvidia.com/ai-enterprise/5.1/user-guide/index.html#preparing-virtual-function-for-sr-iov-vgpu shows that Use only the custom script sriov-manage provided by NVIDIA AI Enterprise for the purpose. | |
| 10:33:25 | pas-ha[m] | yes, as I said, to properly use MIG you need AI Enterprise drivers... | |
| 10:35:47 | r-taketn | I got it. thank you for good advices | |
| 10:38:28 | frickler | sean-k-mooney: tested your eventlet fix and now it seems to hang somewhere, but I'm not sure which test is actually getting stuck https://zuul.opendev.org/t/openstack/build/dde58528ed9841f4b21d0091750fa176 | |
| 10:39:02 | frickler | at least the functional job is passing now | |
| 10:39:15 | sean-k-mooney | so thats the unit test job | |
| 10:39:27 | sean-k-mooney | and i only changed form assert_called_once to assert_called | |
| 10:39:42 | sean-k-mooney | so that not related to the change its somethign deeper | |
| 10:40:54 | sean-k-mooney | frickler: lookign at the logs the tests were still running and passing form what i can see | |
| 10:40:58 | sean-k-mooney | and we just hit the timeout | |
| 10:43:00 | sean-k-mooney | frickler: im not seing anything hang in the logs or even tak a large amoutn of time | |
| 10:46:31 | sean-k-mooney | it took about 3 mins to get to the poitn of runnign tox which does not seem exceeive 4.5 total to run the first test | |
| 10:47:20 | sean-k-mooney | oh i see | |
| 10:47:22 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/dde58528ed9841f4b21d0091750fa176/log/job-output.txt#33835-33836 | |
| 10:47:36 | sean-k-mooney | the absence of logs it just stopped making progress | |
| 10:48:15 | sean-k-mooney | i could just turn off os monkey patching as well | |
| 10:48:57 | sean-k-mooney | what im most concerned about by the way is this breaking oru syncronise decorator | |