| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-19 | |||
| 16:40:28 | bauzas | I think I found why | |
| 16:40:34 | bauzas | 2024-06-19 15:18:23.933 | + ./stack.sh:main:1517 : run_phase stack test-config | |
| 16:40:45 | bauzas | then runs 2024-06-19 15:18:23.963 | ++ functions-common:run_phase:1860 : basename /opt/stack/devstack/extras.d/80-tempest.sh | |
| 16:41:13 | bauzas | which then does some async_wait 2024-06-19 15:18:24.047 | ++ extras.d/80-tempest.sh:source:20 : async_wait install_tempest | |
| 16:41:49 | bauzas | which then installs tempest 2024-06-19 15:18:24.159 | ++ lib/tempest:install_tempest:815 : git_clone https://opendev.org/openstack/tempest.git /opt/stack/tempest master | |
| 16:49:34 | sean-k-mooney | that sounds like a regression in devstack | |
| 16:51:12 | bauzas | maybe I should create the flavors and then configure tempest to use them | |
| 16:52:22 | sean-k-mooney | i have doen that in the past too | |
| 16:52:25 | sean-k-mooney | it works | |
| 16:52:37 | bauzas | do you have some job using it ? | |
| 16:52:38 | sean-k-mooney | thats what i normally do https://review.opendev.org/c/openstack/nova/+/679656/13/playbooks/nfv/nfv.yaml | |
| 16:52:50 | sean-k-mooney | bauzas: not that we merged | |
| 16:53:08 | sean-k-mooney | but i created multi numa testing jobs a few times in the past | |
| 16:53:12 | sean-k-mooney | just to show it workign | |
| 16:53:35 | bauzas | cool | |
| 16:53:52 | bauzas | I already knew how to use another flavor for Tempest https://docs.openstack.org/tempest/latest/configuration.html#flavors | |
| 16:54:20 | bauzas | but I found how to modify the config by a CI job with your change : https://review.opendev.org/c/openstack/nova/+/679656/13/.zuul.yaml | |
| 16:54:22 | bauzas | thanks | |
| 16:54:47 | bauzas | I think it's better than just modifying all of the flavors | |
| 16:55:01 | bauzas | anyway, I'm done too | |
| 16:55:13 | bauzas | I'll change my stuff tomorrow morning | |
| 19:10:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 19:10:38 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 22:43:29 | sean-k-mooney[m] | bauzas: frickler just an fyi i was able to stack on nobe fine by just installing the latest version of setuptool in the requiments and global venv | |
| 22:45:37 | sean-k-mooney[m] | if i get a chance tomorrow ill see if i can tweak devstack to make that work out of the box but i dont think we need to do anything speciall beyond that | |
| 22:47:43 | opendevreview | sean mooney proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 22:51:39 | opendevreview | sean mooney proposed openstack/nova master: Revert "Always delete NVRAM files when deleting instances" https://review.opendev.org/c/openstack/nova/+/922354 | |
| #openstack-nova - 2024-06-20 | |||
| 06:44:57 | gboutry | Hello Nova, I've tried upgrading libvirt/qemu in a dev env, and I'm unable to live migrate anymore. | |
| 06:45:02 | gboutry | I'm getting "virDomainMigrateToURI3() failed" "unsupported configuration: Target CPU feature count 44 does not match source 109". | |
| 06:45:07 | gboutry | The 2 hosts are VMs created the same way, with one having 8 cores and the other one having 4 cores, but same CPU features. | |
| 06:45:16 | gboutry | This error does not appear with libvirt 8.0.0 / QEMU 6.2.0 (ubuntu 22.04 versions), but do appear with libvirt 10.0.0 / QEMU 8.2.1 (ubuntu 24.04 versions). | |
| 06:45:22 | gboutry | These are pretty big version jumps I know, do you have any insight on how to investigate this? | |
| 07:39:27 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 08:25:18 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 08:25:19 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 08:25:19 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 08:25:20 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 08:25:20 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 08:25:21 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 08:25:21 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 08:25:22 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 08:25:22 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 08:25:24 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 08:25:24 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 08:25:26 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 08:25:26 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 08:25:28 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 08:25:28 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 08:25:30 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 08:25:30 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 08:25:32 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 08:25:32 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 08:25:34 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 08:25:34 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 08:25:36 | opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | |
| 08:43:08 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 08:43:09 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 11:03:01 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 11:03:02 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 13:59:42 | frickler | bauzas: did you see the "3 RLock(s) were not greened" messages e.g. at the start of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_42a/922140/13/experimental/nova-mtty/42a251f/controller/logs/screen-n-super-cond.txt ? JayF mentioned something similar yesterday in the ironic channel, not sure if that may actually be relevant? | |
| 14:00:14 | frickler | seems to be related to running on py3.12 at least | |
| 14:04:18 | JayF | I asked my eventlet expert downstream about it. I haven't gotten any response yet. I suspect it maybe that the way they are detecting those locks are getting false positives. | |
| 14:25:40 | frickler | ack, otherwise one may want to insert more debugging output into that warning | |
| 14:48:48 | JayF | There isn't much more to give. Basically it's just inquiring from the garbage collector if it's tracking any rlock objects. It can't really reliably /do/ anything with those objects if detected. | |
| 14:49:16 | JayF | I printed the information on the objects, and nonsurprisingly, it wasn't useful. I suspect it might be possible to get deeper with a debugger + a lower level knowledge of python than I have at hand | |
| 16:28:48 | opendevreview | Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421 | |
| 16:51:47 | melwitt | frickler, JayF: I looked up that log message in eventlet and there's a code comment about how in python < 3.10 the warning won't be logged, so that at least explains why we start seeing it in newer python https://github.com/eventlet/eventlet/blob/ec6c0fffccc129bea468a5badb03034d83f8cc5e/eventlet/patcher.py#L406 | |
| 16:52:10 | JayF | melwitt: well, I think there's a behavior change in 3.12 | |
| 16:52:29 | JayF | melwitt: because chasing down + eliminating those warnings was part of getting eventlet happier when the changes started | |
| 16:53:11 | melwitt | yeah, the code comment explains the behavior change | |
| 16:53:38 | JayF | it explains behavior change from 3.9 -> 3.10, not >=3.10 changing | |
| 16:54:28 | JayF | I'm going to test on 3.10 with my ironic reproducer and see | |
| 16:54:29 | melwitt | "On older Pythons (< 3.10), gc.get_objects() won't return any RLock instances, so this warning won't get logged on older Pythons." that isn't explaining what happens different in >= 3.10? | |
| 16:55:02 | JayF | I'm saying that at no point have I said *3.10* is the point where behavior changed. I'm seeing it in 3.12 and absolutely chased down + elminated many of these dangling locks in Ironic on =>3.10 at some point | |
| 16:55:51 | JayF | so either: 1) some non-python-version behavior changed to give us ungreened locks (really just locks that were made before we monkey patched) or 2) python version is the culprit | |
| 16:56:04 | JayF | I don't know which yet, or even if it matters in reality, but it's not the kinda thing I wanna ignore | |
| 16:56:57 | melwitt | ok, well, my response was to "seems to be related to running on py3.12" and I found the code comment so I thought that was the relation | |
| 16:57:27 | JayF | Confirmed: in Ironic python 3.10 does not give the warning, in python 3.12 it gives the warning with 2 locks | |
| 16:57:37 | JayF | I'll test with 3.11 to ensure it's 3.11 -> 3.12 behavior change as hypothesized | |
| 17:14:49 | JayF | melwitt: frickler: FWIW I've confirmed with some certainty the behavior change is python-version-related and not anything else. With Ironic as the example; python 3.11: no ungreened locks, python 3.12: 2 ungreened locks. Only library difference is pyayncore and setuptools installed on py3.12 and installing those under py3.11 doesn't cause the issue. | |
| 17:15:21 | JayF | if nova is seeing something similar, I may file an upstream eventlet issue | |
| 17:31:30 | frickler | JayF: well the log I linked to above was from bauzas testing nova on Noble, so running with py3.12 too. I'd need to double check that it doesn't happen on py3.11 | |
| 17:34:03 | frickler | hmm, opensearch does find some hits in toy-py311 jobs, too, like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce6/922421/1/check/openstack-tox-py311/ce64a27/job-output.txt | |
| 17:34:18 | JayF | they almost always appear in unit tests | |
| 17:34:22 | sean-k-mooney | frickler: i was able to stack on noble fine yestereday | |
| 17:34:22 | JayF | they are only relevant for real runs | |
| 17:34:28 | sean-k-mooney | i have that runnign in a vm currently | |
| 17:34:36 | sean-k-mooney | but i didn try ot use the deploy openstack | |
| 17:38:29 | sean-k-mooney | frickler: i can boot a vm and do a console log show | |
| 17:39:21 | frickler | JayF: ah, o.k., the number found in those unit tests seems to go from 1 to 4 though from 3.11 to 3.12. also not sure neutron-functional job falls https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cec/917430/8/check/neutron-fullstack-with-uwsgi/cec5565/controller/logs/services.txt | |
| 17:39:37 | frickler | other than that the only hit is the new noble job | |
| 17:41:04 | sean-k-mooney | im sure there are proably bugs but at first glance things appear to work fin on noble and python 3.12 | |
| 17:41:04 | frickler | sean-k-mooney: yes, what triggered me to look at the logs were the tempest failures in the mtty job. but they seem to be related to the actual mtty stuff afaict | |
| 17:41:26 | JayF | My current working hypothesis is it's a reporting issue, not a functional one. | |
| 17:41:46 | sean-k-mooney | frickler: ah ok well i htink there is a bug in sylvains nova patch | |
| 17:41:54 | sean-k-mooney | or somehting related | |
| 17:42:19 | frickler | JayF: yes, at least I have no data that would say otherwise, but would be great if we could be sure of that | |
| 17:42:22 | sean-k-mooney | the tempet test are failign because nova is not reporting an resouce provider inventireos for the mtty devices | |
| 17:42:57 | sean-k-mooney | frickler: the only thing i had to do was pip install -U setuptools in the requiremend and global venv | |
| 17:43:14 | sean-k-mooney | i started on a patch to do that in devstack yesterday but didnt get back to it | |