| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-19 | |||
| 19:10:38 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 22:43:29 | sean-k-mooney[m] | bauzas: frickler just an fyi i was able to stack on nobe fine by just installing the latest version of setuptool in the requiments and global venv | |
| 22:45:37 | sean-k-mooney[m] | if i get a chance tomorrow ill see if i can tweak devstack to make that work out of the box but i dont think we need to do anything speciall beyond that | |
| 22:47:43 | opendevreview | sean mooney proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 22:51:39 | opendevreview | sean mooney proposed openstack/nova master: Revert "Always delete NVRAM files when deleting instances" https://review.opendev.org/c/openstack/nova/+/922354 | |
| #openstack-nova - 2024-06-20 | |||
| 06:44:57 | gboutry | Hello Nova, I've tried upgrading libvirt/qemu in a dev env, and I'm unable to live migrate anymore. | |
| 06:45:02 | gboutry | I'm getting "virDomainMigrateToURI3() failed" "unsupported configuration: Target CPU feature count 44 does not match source 109". | |
| 06:45:07 | gboutry | The 2 hosts are VMs created the same way, with one having 8 cores and the other one having 4 cores, but same CPU features. | |
| 06:45:16 | gboutry | This error does not appear with libvirt 8.0.0 / QEMU 6.2.0 (ubuntu 22.04 versions), but do appear with libvirt 10.0.0 / QEMU 8.2.1 (ubuntu 24.04 versions). | |
| 06:45:22 | gboutry | These are pretty big version jumps I know, do you have any insight on how to investigate this? | |
| 07:39:27 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | |
| 08:25:18 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 08:25:19 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 08:25:19 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 08:25:20 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 08:25:20 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 08:25:21 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 08:25:21 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 08:25:22 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 08:25:22 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 08:25:24 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 08:25:24 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 08:25:26 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 08:25:26 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 08:25:28 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 08:25:28 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 08:25:30 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 08:25:30 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 08:25:32 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 08:25:32 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 08:25:34 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 08:25:34 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 08:25:36 | opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | |
| 08:43:08 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 08:43:09 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 11:03:01 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Enable an experimental job for testing mtty https://review.opendev.org/c/openstack/nova/+/922140 | |
| 11:03:02 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : mtty: add devstack compilation for fedora builds https://review.opendev.org/c/openstack/nova/+/918420 | |
| 13:59:42 | frickler | bauzas: did you see the "3 RLock(s) were not greened" messages e.g. at the start of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_42a/922140/13/experimental/nova-mtty/42a251f/controller/logs/screen-n-super-cond.txt ? JayF mentioned something similar yesterday in the ironic channel, not sure if that may actually be relevant? | |
| 14:00:14 | frickler | seems to be related to running on py3.12 at least | |
| 14:04:18 | JayF | I asked my eventlet expert downstream about it. I haven't gotten any response yet. I suspect it maybe that the way they are detecting those locks are getting false positives. | |
| 14:25:40 | frickler | ack, otherwise one may want to insert more debugging output into that warning | |
| 14:48:48 | JayF | There isn't much more to give. Basically it's just inquiring from the garbage collector if it's tracking any rlock objects. It can't really reliably /do/ anything with those objects if detected. | |
| 14:49:16 | JayF | I printed the information on the objects, and nonsurprisingly, it wasn't useful. I suspect it might be possible to get deeper with a debugger + a lower level knowledge of python than I have at hand | |
| 16:28:48 | opendevreview | Rafael Weingartner proposed openstack/nova master: Block VM instance resize to flavor with different aggregates https://review.opendev.org/c/openstack/nova/+/922421 | |
| 16:51:47 | melwitt | frickler, JayF: I looked up that log message in eventlet and there's a code comment about how in python < 3.10 the warning won't be logged, so that at least explains why we start seeing it in newer python https://github.com/eventlet/eventlet/blob/ec6c0fffccc129bea468a5badb03034d83f8cc5e/eventlet/patcher.py#L406 | |
| 16:52:10 | JayF | melwitt: well, I think there's a behavior change in 3.12 | |
| 16:52:29 | JayF | melwitt: because chasing down + eliminating those warnings was part of getting eventlet happier when the changes started | |
| 16:53:11 | melwitt | yeah, the code comment explains the behavior change | |
| 16:53:38 | JayF | it explains behavior change from 3.9 -> 3.10, not >=3.10 changing | |
| 16:54:28 | JayF | I'm going to test on 3.10 with my ironic reproducer and see | |
| 16:54:29 | melwitt | "On older Pythons (< 3.10), gc.get_objects() won't return any RLock instances, so this warning won't get logged on older Pythons." that isn't explaining what happens different in >= 3.10? | |
| 16:55:02 | JayF | I'm saying that at no point have I said *3.10* is the point where behavior changed. I'm seeing it in 3.12 and absolutely chased down + elminated many of these dangling locks in Ironic on =>3.10 at some point | |
| 16:55:51 | JayF | so either: 1) some non-python-version behavior changed to give us ungreened locks (really just locks that were made before we monkey patched) or 2) python version is the culprit | |
| 16:56:04 | JayF | I don't know which yet, or even if it matters in reality, but it's not the kinda thing I wanna ignore | |
| 16:56:57 | melwitt | ok, well, my response was to "seems to be related to running on py3.12" and I found the code comment so I thought that was the relation | |
| 16:57:27 | JayF | Confirmed: in Ironic python 3.10 does not give the warning, in python 3.12 it gives the warning with 2 locks | |
| 16:57:37 | JayF | I'll test with 3.11 to ensure it's 3.11 -> 3.12 behavior change as hypothesized | |
| 17:14:49 | JayF | melwitt: frickler: FWIW I've confirmed with some certainty the behavior change is python-version-related and not anything else. With Ironic as the example; python 3.11: no ungreened locks, python 3.12: 2 ungreened locks. Only library difference is pyayncore and setuptools installed on py3.12 and installing those under py3.11 doesn't cause the issue. | |
| 17:15:21 | JayF | if nova is seeing something similar, I may file an upstream eventlet issue | |
| 17:31:30 | frickler | JayF: well the log I linked to above was from bauzas testing nova on Noble, so running with py3.12 too. I'd need to double check that it doesn't happen on py3.11 | |
| 17:34:03 | frickler | hmm, opensearch does find some hits in toy-py311 jobs, too, like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce6/922421/1/check/openstack-tox-py311/ce64a27/job-output.txt | |
| 17:34:18 | JayF | they almost always appear in unit tests | |
| 17:34:22 | sean-k-mooney | frickler: i was able to stack on noble fine yestereday | |
| 17:34:22 | JayF | they are only relevant for real runs | |
| 17:34:28 | sean-k-mooney | i have that runnign in a vm currently | |
| 17:34:36 | sean-k-mooney | but i didn try ot use the deploy openstack | |
| 17:38:29 | sean-k-mooney | frickler: i can boot a vm and do a console log show | |
| 17:39:21 | frickler | JayF: ah, o.k., the number found in those unit tests seems to go from 1 to 4 though from 3.11 to 3.12. also not sure neutron-functional job falls https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cec/917430/8/check/neutron-fullstack-with-uwsgi/cec5565/controller/logs/services.txt | |
| 17:39:37 | frickler | other than that the only hit is the new noble job | |
| 17:41:04 | sean-k-mooney | im sure there are proably bugs but at first glance things appear to work fin on noble and python 3.12 | |
| 17:41:04 | frickler | sean-k-mooney: yes, what triggered me to look at the logs were the tempest failures in the mtty job. but they seem to be related to the actual mtty stuff afaict | |
| 17:41:26 | JayF | My current working hypothesis is it's a reporting issue, not a functional one. | |
| 17:41:46 | sean-k-mooney | frickler: ah ok well i htink there is a bug in sylvains nova patch | |
| 17:41:54 | sean-k-mooney | or somehting related | |
| 17:42:19 | frickler | JayF: yes, at least I have no data that would say otherwise, but would be great if we could be sure of that | |
| 17:42:22 | sean-k-mooney | the tempet test are failign because nova is not reporting an resouce provider inventireos for the mtty devices | |
| 17:42:57 | sean-k-mooney | frickler: the only thing i had to do was pip install -U setuptools in the requiremend and global venv | |
| 17:43:14 | sean-k-mooney | i started on a patch to do that in devstack yesterday but didnt get back to it | |
| 17:43:41 | sean-k-mooney | has anyoen proposed a noble job to the devstack repo yet? | |
| 17:43:58 | sean-k-mooney | if not i can maybe do that this week | |
| 17:45:17 | frickler | ah, yes, I still wanted to add some unit test for edit-constraints to the reqs fix | |
| 17:49:25 | opendevreview | sean mooney proposed openstack/nova master: [WIP] split monkey_patching form import https://review.opendev.org/c/openstack/nova/+/922425 | |
| 18:08:54 | sean-k-mooney | frickler: JayF https://review.opendev.org/c/openstack/devstack/+/922427 should work? | |
| 18:09:21 | JayF | nice, I can hook that up to ironic once it shows it's working elsewhere | |
| 18:09:25 | JayF | would be interesting to see what's busted | |
| 18:09:44 | sean-k-mooney | maybe | |
| 18:09:59 | sean-k-mooney | i just got back to the eventlet stuff after 4 weeks or so | |
| 18:10:15 | sean-k-mooney | so right now im trying to load back up the context on what i was doing | |
| 18:12:47 | sean-k-mooney | JayF: ill generally be moving most of my dev envs over to noble or c10s over the next few weeks, both will be using python 3.12 | |
| 18:13:13 | JayF | Honestly, it's just a fluke I caught this. 3.12 is my default python, I was testing a local dev thing for Ironic, and just saw it | |
| 18:13:48 | sean-k-mooney | i feel like that how we catch most things | |
| 18:14:09 | sean-k-mooney | one of use runs a ill advicsed setup (i.e. the latest and gratest of whatever) and fine odd bugs | |
| 18:15:12 | sean-k-mooney | in anycase that should give use a full tempest-full run on noble without any other chagnes if it stacks properly | |
| 18:15:27 | frickler | sean-k-mooney: while just testing you may also want to drop all the other jobs in the check pipeline | |
| 18:15:29 | JayF | I was about to make some snarky comment about being offended you find my gentoo laptop ill advised ... then I realized you're *probably* actually right :D | |
| 18:15:31 | sean-k-mooney | whihc shoudl be antoher datapoint to if this is an actul problem or jsut a logging issue | |
| 18:15:47 | sean-k-mooney | frickler: yep i remmebered that after i pushed | |
| 18:15:55 | sean-k-mooney | frickler: ill do it if i respin | |
| 18:16:04 | frickler | ack | |
| 18:19:11 | sean-k-mooney | pbr still depends on distutils by the way | |