| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-17 | |||
| 13:54:40 | openstackgerrit | Merged openstack/nova master: Fix NoneType error in _notify_volume_usage_detach https://review.openstack.org/609518 | |
| 13:54:50 | mriedem | tssurya: you want to backport this to rocky? https://review.openstack.org/#/c/611094/ | |
| 13:55:37 | tssurya | mriedem: yea, doing it | |
| 13:55:58 | mriedem | artom: i see you wasted no time https://review.openstack.org/#/q/Icb0bdaf454935b3713c35339394d260b33520de5 | |
| 13:56:42 | dansmith | mriedem: we have some pressure on that one | |
| 13:56:49 | dansmith | rocky should be okay now yeah? | |
| 13:56:54 | tssurya | yea | |
| 13:57:52 | mriedem | dansmith: i just +2d it | |
| 13:58:34 | dansmith | ah was just looking at the conflict, | |
| 13:58:38 | dansmith | but I see you called it out | |
| 14:03:03 | mriedem | anyone know if tripleo is running nova-status upgrade check yet? | |
| 14:03:07 | mriedem | either on install and/or upgrade? | |
| 14:04:34 | mriedem | looks like only OSA and kolla-ansible so far | |
| 14:08:07 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Fix NoneType error in _notify_volume_usage_detach https://review.openstack.org/611326 | |
| 14:09:19 | openstackgerrit | Surya Seetharaman proposed openstack/nova stable/rocky: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611327 | |
| 14:11:15 | bauzas | mriedem: lyarwood is on PTO for the next two weeks, but lemme look | |
| 14:16:46 | tssurya | if any cores have some time (other than mriedem and efried): would be good to have this in https://review.openstack.org/#/c/611162/ | |
| 14:17:03 | mriedem | i was just going to lobby for that :) | |
| 14:17:06 | mriedem | since it's an upgrade impact | |
| 14:17:19 | tssurya | :) | |
| 14:21:19 | bauzas | tssurya: mriedem: mmm, is that only an Ironic thing ? | |
| 14:21:40 | bauzas | mriedem: I was looking for like 2 weeks why some ComputeNode UUID was changing from a libvirt node | |
| 14:21:53 | tssurya | its an only ironic driver thing | |
| 14:21:55 | bauzas | anyway, +Wing | |
| 14:22:01 | tssurya | thanks bauzas | |
| 14:22:42 | bauzas | tssurya: mriedem: can't see a stable/rocky one, please ping me the change once done | |
| 14:22:52 | tssurya | yea proposing one now | |
| 14:22:59 | bauzas | of course, I'll +2 it once the master lands | |
| 14:24:15 | mriedem | bauzas: this is a fun reqspec regression thing too https://review.openstack.org/#/q/topic:bug/1797580+(status:open+OR+status:merged) | |
| 14:24:27 | mriedem | goes back to queens | |
| 14:24:35 | bauzas | mriedem: after 2 weeks of internal bug triage, I love you | |
| 14:24:43 | bauzas | because you help me doing other stuff | |
| 14:24:49 | tssurya | yea we hit that too the marker stuff :( | |
| 14:24:56 | mriedem | ^ isn't marker stuff | |
| 14:25:02 | bauzas | oh this one | |
| 14:25:07 | bauzas | the requested_destination field | |
| 14:25:21 | mriedem | tssurya: https://review.openstack.org/#/c/610974/ | |
| 14:25:51 | tssurya | ah hehe yea | |
| 14:25:53 | tssurya | sorry | |
| 14:26:20 | mriedem | but yeah would be good to get that in also since it affects upgrades | |
| 14:26:24 | mriedem | johnthetubaguy: are you around? | |
| 14:26:58 | openstackgerrit | Surya Seetharaman proposed openstack/nova stable/rocky: Ignore uuid if already set in ComputeNode.update_from_virt_driver https://review.openstack.org/611337 | |
| 14:27:09 | artom | mriedem, yeah, like I said, downstream pressure :) | |
| 14:27:25 | tssurya | bauzas, efried: ^ the backport to rocky for after the master merges | |
| 14:28:11 | mriedem | +1 | |
| 14:34:26 | stephenfin | Can someone help me understand what's going on here? paste.openstack.org/show/732338/ | |
| 14:34:56 | stephenfin | Was discussing it with bauzas earlier today. We've configured 'cpu_allocation=1' and yet it still seems like overallocation is happening | |
| 14:36:04 | bauzas | stephenfin: like I said to you, we should check the instances number | |
| 14:36:49 | stephenfin | bauzas: Yeah, there's one instance on that host before we do the host-evacuate. Once we do that, two instances land on it | |
| 14:37:09 | stephenfin | One instance has 2 vcpu, the other has 1 vcpu | |
| 14:37:12 | bauzas | given vcpus_used is generated by looking up the instances | |
| 14:37:24 | bauzas | host-evacuate is racy | |
| 14:37:44 | bauzas | you should really avoid it, and do two evacuations concurrently | |
| 14:37:57 | bauzas | I thought we commented that btw. | |
| 14:38:21 | stephenfin | Yup, vcpus_used appears to be correct. Compare lines 24-29 to 90-95 | |
| 14:38:42 | bauzas | https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/contrib/host_evacuate.py | |
| 14:39:04 | stephenfin | The second hypervisor (dcfb3fb7) goes from 5/6 used to 7/6 | |
| 14:41:02 | stephenfin | bauzas: But its done client side. It seems odd that the scheduler wouldn't honour overcommit ratios just because the requests came in at the same time. Is this known behavior? | |
| 14:42:13 | mriedem | is the target host in an aggregate with an allocation ratio metadata? | |
| 14:42:38 | stephenfin | sec - checking | |
| 14:42:56 | mriedem | compute-0.localdomain was forced down but was the service actually stopped? | |
| 14:44:31 | bauzas | otp but looking | |
| 14:44:52 | mriedem | because vcpus_used didn't change on compute-0 after the evacuation | |
| 14:45:22 | mriedem | though i guess i wouldn't expect them to until compute-0 is restarted and removes locally evacuated instances | |
| 14:45:42 | mriedem | but not really safe to have the guest running on the source and dest at the same time... | |
| 14:45:48 | stephenfin | I don't see vcpus_used reported for compute-0 in those logs. Mixing it up with compute-1? | |
| 14:46:05 | mriedem | oh yeah you're right | |
| 14:46:29 | mriedem | pi | |
| 14:46:41 | mriedem | you're just wondering why they are evacuated to compute-2 rather than compute-0? | |
| 14:46:47 | mriedem | or not spread between them? | |
| 14:46:58 | stephenfin | spread between compute-1 and compute-2 | |
| 14:47:07 | stephenfin | Well | |
| 14:47:26 | mriedem | right you expect compute_0_instance_1_vcpu to go to compute-1 and compute_0_instance_2_vcpu to go to compute-2 | |
| 14:47:34 | mriedem | which would fill up each of those computes | |
| 14:47:37 | stephenfin | More that the second migration wouldn't fail because it would bump compute-2 over our overallocation ratio | |
| 14:47:45 | stephenfin | Sure, though the opposite could also happen | |
| 14:47:58 | mriedem | well, which release is this? is the CoreFilter enabled? | |
| 14:48:00 | awaugama | I'm the one who tried it, the hosts were not in aggregates | |
| 14:48:14 | awaugama | I can retry with that | |
| 14:48:14 | stephenfin | so compute_0_instance_1_vcpu goes to compute-2 and then compute_0_instance_2_vcpu gets rejected | |
| 14:48:48 | mriedem | if the CoreFilter is not enabled, there will be no cpu limits sent to the compute to claim in the RT | |
| 14:48:54 | mriedem | which means you can overcommit | |
| 14:49:21 | mriedem | if this is pike+, the filter scheduler should be claiming allocations in placement and we don't need to care about vcpu claims in the RT | |
| 14:49:34 | mriedem | so i'd take a look at your resource provider inventory/allocations | |
| 14:49:45 | mriedem | using the osc-placement commands | |
| 14:50:05 | mriedem | https://docs.openstack.org/osc-placement/latest/ | |
| 14:50:21 | awaugama | ok thanks | |
| 14:52:15 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Handle missing marker during online data migration https://review.openstack.org/611343 | |
| 14:56:57 | awaugama | mriedem: sorry I'm confused about something. you're saying placement has it's own ratios that will cause the configured compute node ratios to be ignored? | |
| 14:57:25 | dansmith | awaugama: nova's ratios are mostly irrelevant now, yes | |
| 14:58:13 | awaugama | so that's where the corefilter comes in? | |
| 14:58:51 | mriedem | well, | |
| 14:58:57 | mriedem | is the CoreFilter enabled? it shouldn't be. | |
| 14:59:07 | mriedem | if you're >=pike and using the filter scheduler | |
| 14:59:14 | mriedem | no one told me what release this is | |
| 14:59:27 | awaugama | Sorry, this is Rocky | |
| 14:59:49 | mriedem | the compute service will create a resource provider in the placement service and report it's inventory (vcpu/ram/disk) and allocation ratios for each of those resource classes | |
| 15:00:08 | mriedem | the scheduler will atomically allocate resources from each provider during the evacuate | |
| 15:00:25 | mriedem | so it should only overcommit if the VCPU inventory allocation_ratio in placement is > 1.0 | |
| 15:00:58 | mriedem | so, find the uuid for the compute-2 resource provider using "openstack resource provider list" | |