Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-09-27
16:50:58 justas_napa Yes, I understand the issue of cross-project dependancy
16:51:14 gibi bauzas: if the former, then the a DNM nova patch on top of the nova series could configure zuul to use the later method + add a Depends-On to the os-vig patch
16:51:18 justas_napa and that our change is not atomic across the projects
16:51:24 bauzas justas_napa: but here, the question is that for some libraries, we maybe use a specific release and not using master
16:51:52 bauzas gibi: yeah, that would work
16:52:06 bauzas justas_napa: do you understand what gibi is explaining ?
16:52:08 justas_napa OK, I understand the issue, but I'm not sure I can offer a solution.
16:52:44 bauzas justas_napa: well, we maybe have a solution, that's the point
16:52:45 justas_napa One way is to use our ci/cd setup where we runn all regression tests and have control of releases of all projects
16:53:21 bauzas anyway, we should be discussing this in the spec
16:53:43 bauzas justas_napa: are you saying you could provide a third-party CI for nova/neutron ? :D
16:53:58 bauzas this would be lovely
16:54:16 justas_napa not sure what you mean 3rd party
16:54:44 justas_napa but we have our own ci/cd pipeline running integration tests with our changes
16:54:55 bauzas ok, then we'll discuss this in the spec and we can also provide you some links if you want
16:55:09 justas_napa to ensure we do not break anything with our OS patches
16:55:19 bauzas justas_napa: are you running tempest tests ?
16:55:22 justas_napa yes
16:55:44 bauzas cool, then that's something we need to discuss then
16:55:55 bauzas anyway, I guess we're done about your topic
16:56:01 bauzas we'll continue discussing
16:56:21 justas_napa sure, thanks for all questions and comments
16:56:25 bauzas folks, any other item to bring before we end the meeting ?
16:56:58 bauzas looks not
16:57:01 bauzas thanks all
16:57:04 bauzas #endmeeting
16:57:04 opendevmeet Meeting ended Tue Sep 27 16:57:04 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:57:04 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-09-27-16.00.html
16:57:04 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-09-27-16.00.txt
16:57:04 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-09-27-16.00.log.html
16:57:08 elodilles thanks o/
16:58:35 gibi o/
18:45:22 JayF bauzas: o/ We need to meet up and discuss timing for joint Ironic<>Nova PTG session.
18:45:42 JayF bauzas: unsure what timezone you're in, but if you're asleep/AFK right now, just DM me and we can figure it out async
18:59:56 stephenfin JayF: bauzas is in France so CET. You'll catch him tomorrow
19:00:22 JayF ack; I figured, not a big deal, took me a couple of days to voluntell the right people I needed them in that session and get scheduling deets :D
#openstack-nova - 2022-09-28
06:10:44 opendevreview Amit Uniyal proposed openstack/nova master: Adds check for VM snapshot fail while quiesce https://review.opendev.org/c/openstack/nova/+/852171
07:00:18 bauzas JayF: ping me when you're back if you want, sure
10:10:38 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Bump min oslo.concurrencty to >= 5.0.1 https://review.opendev.org/c/openstack/nova/+/859421
10:29:10 damiandabrowski hey folks, I noticed that if compute node already exceeds its overcommit ratios, it's not possible to migrate VMs out of it
10:29:29 damiandabrowski it makes no sense for me and it should be considered as a bug, but maybe there is some reason behind it?
10:29:31 damiandabrowski https://paste.openstack.org/raw/byjHufprWu3hwDFPSKAD/
10:37:37 gibi damiandabrowski: the reason is that if you the compute is already exceeds overcommit then it cannot have new allocations. And the migration needs to move the instance allocation to the migration_uuid in placement and that would require a new allocation even if the total usage would not change.
10:37:54 gibi what you can do is
10:38:19 gibi i) increase the allocation ratio temporarily to the level where the usage not exceeds it
10:38:25 gibi ii) migrate the workload out
10:38:41 gibi iii) revert the allocation ratio to the original level
10:39:27 damiandabrowski thanks, i'll do that
10:39:31 damiandabrowski so you don't consider it as a bug, right?
10:40:27 gibi you exceeded the overallocation so you are in a not supported state I would say
10:41:28 damiandabrowski okok, thanks for clarification
12:17:52 opendevreview Oleksii Butenko proposed openstack/os-vif master: Add new os-vif `network` property https://review.opendev.org/c/openstack/os-vif/+/859574
12:19:53 jkulik damiandabrowski: FYI, we have a patch downstream in our Placement to allow switching resources on overused providers, so migrations off should still be possible: https://github.com/sapcc/placement/commit/2691056d6fa3e7db0cf9966b082af51ae6b5dda9
12:20:29 jkulik use at your own risk, though
12:29:45 damiandabrowski oh that's convenient, thanks
12:41:51 opendevreview Oleksii Butenko proposed openstack/nova master: Napatech SmartNIC support https://review.opendev.org/c/openstack/nova/+/859577
12:53:11 opendevreview J.P.Klippel proposed openstack/nova master: Fix link to Cyborg device profiles API https://review.opendev.org/c/openstack/nova/+/859578
13:17:40 opendevreview Justas Poderys proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290
13:18:57 justas_napa Hi bauzas, gibi. Re: Napatech smartnic support, we have now pushed the code changes to opendev
13:19:22 justas_napa all changes are visible in opendev via https://review.opendev.org/q/topic:Napatech_SmartNIC_support
13:45:05 noonedeadpunk gibi: I guess the question is - why new allocation is created at the first place. As it's already present, isn't it?
13:46:20 noonedeadpunk so why another allocation is being created against source host
13:47:36 gibi noonedeadpunk: it is a technicality. When the instance is created the resource allocation of the instance is held by the consumer=instance.uuid in Placemnet. When the instance is being migrated, Nova first moves the allocation held by the consumer=instance.uuid to consumer=migration.uuid in Placement, then nova will call the scheduler to select a target host for the migration and allocate the
13:47:42 gibi resource on the target host to consumer=instance.uuid in Placement.
13:48:21 gibi noonedeadpunk: the move of the allocation from instance.uuid to migration.uuid is the one that considered as a new allocation in Placement and rejected as the host is already overallocated
13:48:50 noonedeadpunk well, it's quite slight line between bug and "as designed"
13:49:40 noonedeadpunk because for me - allocation change that does not involve change of resources shouldn't be considered as new one...
13:49:44 gibi I intentionally not say it is a bug or not a bug, what I say is that having the host overallocated (over the overallocation limit) is the cause of the problem. A host should never be in that state
13:50:54 noonedeadpunk but I do agree that workaround is easy enough to not spend time on making placement logic mroe complex for this
13:51:12 gibi in all these cases you have to first figure out why and how you ended up in an overallocated state
13:51:36 gibi and as you said recovering from this state is not as hard
13:52:37 noonedeadpunk I can say super simply example - you decided that overcommit ratio is too high and it's good to lower it down. And you have prometheus that can tell ou on average for region overcommit is lower then you want to define
13:53:24 noonedeadpunk So you lower it down, but some computes are still having higher overcommit, as you have not checked that against placement per host
13:53:46 noonedeadpunk And you can't lower it down easily by reverting setting and disabling compute for scheduling on top
13:55:17 noonedeadpunk * And you can't lower it down easily by migrating instances out so you have to revert setting
13:59:58 gibi basically that process makes the mistake of lowering the allocation ratio on the host before checking it if it causes overallocation
14:01:41 gibi so the proper process would be (in my eyes): 1) disable the host temporarily to avoid new scheduling while you reconfigure it 2) migrate VMs out from the host to achive the target resource usage 3) reconfigure the host with the new allocation ratio 4) enable the host
14:03:00 noonedeadpunk well... it's hard to disagree here :)
14:03:34 noonedeadpunk (it's more tricky to do though with microversion 2.88)
14:07:51 noonedeadpunk or maybe not and it's jsut me who got used to deal with nova api...
14:08:36 noonedeadpunk as eventually representation of inventory in placement is way more accurate
14:10:12 gibi yeah you should use placement to calculate how many vms you need to move
14:29:26 auniyal Hello #openstack-nova
14:29:42 noonedeadpunk gibi: the problem is that with sdk it's quite tricky to use placement as you need to really call api rather then get resources with all attributes in it (like it was with hypervisors)
14:30:07 auniyal how this works - https://opendev.org/openstack/nova/src/commit/aad31e6ba489f720f5bdc765c132fd0f059a0329/nova/context.py#L396
14:55:54 noonedeadpunk gibi: just to express my pain as operator. https://paste.openstack.org/show/bpA2Iq28mHEfo8r4sH2W/ consumes about 3 seconds to execute. And as you see - overrides microversion <2.88.
14:56:13 noonedeadpunk If I am to use palcement to get the same data - I will need code like that https://paste.openstack.org/show/bOsJwmnMMpftgZ01St0o/ and it takes... 42 seconds to execute against exact same cluster
14:57:28 noonedeadpunk And not saying about load on APIs as for each resource provider I will need to make 2 calls to placement
14:59:41 noonedeadpunk so for users deprecation of providing resource statistics from hypervisors is quite serious regression in performance. And if I want to monitor that regulary - it's really waste of time.
15:00:02 gibi noonedeadpunk: interesting. so there is some heavy inefficiencies somewhere as it just 2x the number of calls but it takes more than 10 times the time to execute
15:00:49 noonedeadpunk fwiw - I was testing remote cloud (not from inside of it)
15:03:28 noonedeadpunk just to show that I'm not exaggerating https://paste.openstack.org/show/b1irA4GLwJEdUORuLarb/
15:04:40 noonedeadpunk But I'm not sure about 2x number of calls. As it feels that from hypervisor data was fetched with single api call...
15:04:47 noonedeadpunk Not sure though
15:05:04 noonedeadpunk (or at least I can't explain it otherwise)
15:06:31 gibi hm, you are right the hypervisor one returned all compute from the cluster
15:06:34 gibi /os-hypervisors/detail
15:06:46 gibi so then I think it would make sense to add a similar api for placement
15:06:55 gibi single call that returns all the usages per provider
15:07:28 gibi this explains the preformance differences

Earlier   Later