| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-12-15 | |||
| 14:12:49 | johnthetubaguy | bauzas: I am sort of around today if there are questions on the spec, in case that helps. | |
| 14:15:58 | gibi | sean-k-mooney: never tried backup | |
| 14:16:35 | johnthetubaguy | sean-k-mooney: I thought it was just a regular snapshot using the users token? Its not on a schedule, we just delete old ones after creating a new one... I thought. Been about 5 years since I tried it mind. Although those glance calls can also get a service token attached, to stop the user token expiry issues. | |
| 14:17:35 | johnthetubaguy | you could require a service token, a bit like how we discussed for using user tokens for port binding in neutron | |
| 14:42:24 | opendevreview | Merged openstack/python-novaclient master: tests: Fix Python 3.11 compatibility https://review.opendev.org/c/openstack/python-novaclient/+/867270 | |
| 14:43:28 | stephenfin | sean-k-mooney: yeah, exactly. What's the RFE/bug? | |
| 15:00:18 | opendevreview | Pierre-Samuel Le Stang proposed openstack/nova master: Reproducer test of bug #1999674 https://review.opendev.org/c/openstack/nova/+/867807 | |
| 15:11:44 | bauzas | johnthetubaguy: sean-k-mooney: fwiw, I'm on the spec since 30 mins | |
| 15:11:51 | bauzas | should be done in 10 mins | |
| 15:12:41 | sean-k-mooney | ack | |
| 15:23:12 | bauzas | johnthetubaguy: sean-k-mooney send to the gate with comments | |
| 15:23:16 | bauzas | nothing important | |
| 15:23:29 | bauzas | and as said, we can continue discussing about those during the implementation | |
| 15:23:36 | bauzas | but I don't wanna hold this spec for this cycle | |
| 15:35:11 | opendevreview | Merged openstack/nova-specs master: Ironic shard_key to replace peer_list https://review.opendev.org/c/openstack/nova-specs/+/862833 | |
| 16:02:28 | rloo | hi nova-folks. thanks johnthetubaguy for fixing https://bugs.launchpad.net/nova/+bug/1974070 because I need that fix in wallaby! Is this https://review.opendev.org/c/openstack/nova/+/864773 backportable, it added a new config option? | |
| 16:05:06 | johnthetubaguy | rloo: your very welcome! I think it is backportable, because its a workaround config option, at least I think that is what we agreed. I haven't had chance to propose it myself though. | |
| 16:06:22 | rloo | thx johnthetubaguy ! if i can recall how to, i'll try to propose them and see how that goes :) | |
| 16:07:28 | rloo | btw johnthetubaguy, this didn't get merged, is it still desired? https://review.opendev.org/c/openstack/nova/+/842478 | |
| 16:09:06 | opendevreview | Ruby Loo proposed openstack/nova stable/zed: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867642 | |
| 16:17:46 | opendevreview | Konrad Gube proposed openstack/nova-specs master: Use extend volume completion action https://review.opendev.org/c/openstack/nova-specs/+/855490 | |
| 16:24:57 | opendevreview | Konrad Gube proposed openstack/nova-specs master: Use extend volume completion action https://review.opendev.org/c/openstack/nova-specs/+/855490 | |
| 16:33:38 | johnthetubaguy | rloo: good point, that is still needed for cases like when you mark and available node as in-maintenance but it gets picked before nova updates placement. we shouldn't need it work around automatic cleaning anymore though, so its more of an edge case now | |
| 16:33:45 | opendevreview | John Garbutt proposed openstack/nova master: Ironic: retry when node not available https://review.opendev.org/c/openstack/nova/+/842478 | |
| 16:35:41 | rloo | yes, i could see that being useful but am really interested (now anyway!) in the fix for cleaning. Thx johnthetubaguy, will keep an eye out for that one too if i remember. | |
| 16:37:46 | johnthetubaguy | I kept running out of candidates that worked with the other fix, in the specific scenario where I was hitting the cleaning race, hence the better fix, kinda glad it didn't work first time, its a much better fix :) | |
| 16:41:12 | rloo | definitely! | |
| 17:00:07 | clarkb | bauzas: to followup on nova tox v4 compat my DNM change looked clean except for the openstacksdk functional job which is actually an openstacksdk tox.ini problem. I've pushed https://review.opendev.org/c/openstack/openstacksdk/+/867827 to address that (not sure if it is complete yet) | |
| 17:03:11 | bauzas | on a meeting but ack | |
| 17:35:57 | opendevreview | Pierre-Samuel Le Stang proposed openstack/nova master: Correctly reset instance task state in rebooting hard https://review.opendev.org/c/openstack/nova/+/867832 | |
| 18:21:24 | sean-k-mooney | rloo: johnthetubaguy yes that shoudl be backporatble | |
| 18:22:04 | sean-k-mooney | in fact since https://review.opendev.org/c/openstack/nova/+/867642 is proposed i might as well review it now | |
| 18:23:13 | rloo | thx sean-k-mooney! I got a conflict when i tried to cherry pick to yoga, haven't yet had time to look at that. | |
| 18:23:58 | sean-k-mooney | ack ill be around tomorrow as well but after that ill be back in the new year | |
| 18:24:15 | sean-k-mooney | feel free to add me to them or ping me and ill happliy review the backports | |
| 18:24:44 | rloo | Thanks sean-k-mooney!! | |
| 20:14:39 | opendevreview | Merged openstack/nova master: Ironic: retry when node not available https://review.opendev.org/c/openstack/nova/+/842478 | |
| 21:16:23 | opendevreview | Ruby Loo proposed openstack/nova stable/yoga: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867912 | |
| 22:41:03 | opendevreview | Ruby Loo proposed openstack/nova stable/yoga: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867912 | |
| 23:11:27 | opendevreview | Ruby Loo proposed openstack/nova stable/yoga: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867912 | |
| 23:14:13 | opendevreview | Ruby Loo proposed openstack/nova stable/xena: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867338 | |
| 23:23:26 | opendevreview | Ruby Loo proposed openstack/nova stable/xena: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867913 | |
| 23:24:16 | opendevreview | Ruby Loo proposed openstack/nova stable/wallaby: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867914 | |
| #openstack-nova - 2022-12-16 | |||
| 05:04:49 | thuvh | IDENTIFY | |
| 08:08:17 | congnt | Hi everyone, I have a new compute with CPU (Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz) Icelake Intel. But libvirt recognize model is Broadwell-noTSX-IBRS. Anyone know this bug? I'm use OpenStack Victoria deploy by kolla-ansible, libvirt version 6.0.0-0ubuntu8.8 | |
| 08:15:25 | obre | congnt: Hi! I guess that a "cat /proc/cpuinfo | grep mpx" returnes no results? | |
| 08:15:44 | congnt | Yes | |
| 08:16:07 | congnt | No result with this command | |
| 08:17:08 | obre | The issue is basicly that libvirt expects that flag to call a CPU "IceLake". | |
| 08:17:33 | obre | But Intel does not ship mpx-support for quite a few of their newer CPU-lines. | |
| 08:17:51 | obre | But you can configure libvirt with your own custom cpu-models. | |
| 08:18:21 | obre | We do for example use this file on our IceLake nodes: https://github.com/ntnusky/profile/blob/master/files/libvirt/cpu/x86_Icelake-Server-NTNU.xml | |
| 08:19:40 | obre | Add that file to /usr/share/libvirt/cpu_map/ and update /usr/share/libvirt/cpu_map/index.xml to include a link to it. Then after a restart of libvirt you should have an ICE-Lake CPU available. | |
| 08:32:25 | congnt | obre: thank you so much, i will research it. | |
| 10:52:59 | sean-k-mooney | congnt: ya so this is a common issue with older release of libvirt. | |
| 10:54:32 | sean-k-mooney | congnt: there are 3 ways to work around it if you cant us a newer libvirt, 1 is create a custom model, possibly by copying the one form a newer release, the next is to use the model that matches closed by setting cpu_mode=custom and cpu_model=<whatever> the final way to work around this is set cpu_mode=host-passthrough | |
| 10:55:24 | sean-k-mooney | for 2 i forgot to say you can add the missing cpu flage with cpu_model_extra_flags | |
| 10:55:35 | sean-k-mooney | those config options are all in the libvirt section fo the nova.conf | |
| 11:09:39 | songwenping | sean-k-mooney: hi, i remember the live migration will check source node's resource capacity right? | |
| 11:20:09 | songwenping | sean-k-mooney: hi, i remember the live migration will also check source node's memory capacity right? | |
| 11:27:22 | sean-k-mooney | i would have to check if live migration is using 2 seperate allcotions or just one. i know for some move operations we use the migration context to hold the allcoation for one of the nodes and the vm for the other | |
| 11:28:28 | sean-k-mooney | we have not ported all move operations to use that workflows. cold migration does if i recall correctly, evacuate does not i think live migration will use 2 allocations but again i would need to look at the code to confirm | |
| 11:32:46 | sean-k-mooney | gibi: bauzas do ye recall off the top of ye're heads what we do for live migration ^ | |
| 11:56:48 | gibi | live migration uses the migration uuid to hold the source node alloc | |
| 11:56:59 | gibi | only evacuate is an exception | |
| 11:59:06 | sean-k-mooney | ack that is what i tought | |
| 11:59:43 | sean-k-mooney | we should fix evacuation sooner rather then later... im going to go add that to our downstream backlog | |
| 12:00:33 | sean-k-mooney | maybe i can push for that in B or C it would be nice to get that finally fixed | |
| 12:00:49 | sean-k-mooney | although the placement allocation explostion issue might be more imporant | |
| 12:01:17 | sean-k-mooney | we also still need to start using the consumer types feature right | |
| 12:01:32 | sean-k-mooney | to actully start marging the migration allcoations as migrations | |
| 12:02:02 | gibi | yes and yes | |
| 12:03:26 | sean-k-mooney | by the way i will shortly be resuming the pci series review stephenfin are you on pto from today? | |
| 12:04:14 | sean-k-mooney | stephenfin: im hoping to finish reviewign the rest of the pci seriese today but if not it will be my goal to get it completed the first week of january when im back | |
| 12:42:49 | songwenping | sean-k-mooney,gibi: it seems not reasonable if live migration uses 2 seperate allocations, the vm cannot be migrate if the source node have not enough resouce. | |
| 12:50:46 | sean-k-mooney | songwenping: it can we wont stop the migration if the source is over commited | |
| 12:51:20 | sean-k-mooney | that will fail for evacuate but live migration shoudl work | |
| 12:51:47 | sean-k-mooney | that said you shoudl never get into that situration unless you change something in a way that was unsupproted or hit a bug | |
| 12:52:13 | sean-k-mooney | for example reduced the memory in the source node either intentiollay or due to a dim failure or change the allcoation ratio | |
| 12:52:50 | songwenping | we use the Rocky code, and placement is integrated with nova. | |
| 12:53:17 | sean-k-mooney | multiple allocation was intoduced in Queens | |
| 12:53:23 | sean-k-mooney | so it shoudl be there in rocky | |
| 12:54:16 | songwenping | but we encounter the problem, the vm cannot migrate because the source node's memory over commit | |
| 12:55:12 | sean-k-mooney | we may have a bug in rocky or master then but you shoudl not be in an over commit senario | |
| 12:55:36 | sean-k-mooney | its fine to oversubscibe provided the total amoutn adn allocation ratio align | |
| 12:55:56 | sean-k-mooney | the temporay fix woudl be to increase the allocation ratio to enabel the migration | |
| 12:56:03 | sean-k-mooney | and then restore it when done | |
| 12:56:16 | sean-k-mooney | althernitivly you coudl try cold migration. | |
| 12:58:40 | songwenping | yes, we try to increase the allocation ratio now, and analyse why the memory is over commit | |
| 13:12:26 | gibi | yepp it is a know behavior that you cannot migrate from an overallocated node. remove the overallocation by temporarily increasing allocation ratio, move the instance, restore the allocation ratio. And separately investigate how you ended up in an overallocated scenario | |
| 13:18:45 | sean-k-mooney | gibi: i tought that only affected migrtions that uses a single allcoation | |
| 13:20:25 | gibi | sean-k-mooney: I think it is the other way around. Evacuation is the only move that does not use migration allocation on the source. It only extend the instance allocation to cover both the source and the dest node. So I think evac is not effect by this | |
| 13:20:49 | gibi | all the other move operators move the instance allocation from the instance_uuid to the migration_uuid on the source node | |
| 13:21:01 | gibi | that move allocation is what triggers the situation | |
| 13:21:10 | gibi | becuase placement does not have a move semantic | |
| 13:22:38 | sean-k-mooney | hum maybe | |
| 13:23:03 | gibi | https://github.com/openstack/nova/blob/36091a7ed7ad553d5cbb5dcfde0090e1e762bc34/nova/scheduler/client/report.py#L2023-L2043 | |
| 13:23:04 | sean-k-mooney | evacuate is broken for overcommited case too however | |
| 13:23:23 | sean-k-mooney | the extention failes if the souce is over allcoated | |
| 13:23:58 | gibi | OK then I was mistaken on that part. then all allocation manipulation is reject by placement if there is at least one overallocated RP in the change | |