Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-02-04
19:31:41 alexe9191 yes one moment let me sanitise it
19:32:13 sean-k-mooney ya if you have enable the plamcent version you can also disable the az filter
19:32:31 alexe9191 indeed that will also be done since it can be used
19:32:36 alexe9191 http://paste.openstack.org/show/802341/
19:32:36 alexe9191 so here is the version that works:
19:32:37 sean-k-mooney over the release we have slowly been moving things too placment where it makes sense
19:33:15 sean-k-mooney yep so you went form 148 down to 104
19:33:25 sean-k-mooney then those would get weighed
19:33:58 alexe9191 http://paste.openstack.org/show/802342/ this is the one that does not (COmpute is reporting 0 cause those are disabled)
19:34:20 alexe9191 Filter AvailabilityZoneFilter returned 8 because 8 are only in zone1
19:34:28 alexe9191 so I am starting with less than I should
19:34:57 sean-k-mooney so the first following filters RetryFilter AvailabilityZoneFilter AggregateDiskFilter, AggregateCoreFilter and AggregateRamFilter can be removed
19:35:07 sean-k-mooney and NUMATopologyFilter should come last
19:36:07 sean-k-mooney so the one that faild had most of the host elimiated by the AvailabilityZoneFilter
19:36:29 alexe9191 indeed, because none of the hosts in zone1 reported it self to placement to be a good match for that flavor
19:36:40 alexe9191 though there are plenty of space on those hosts to cover the needed resources
19:36:47 alexe9191 and I have no max placement in the config
19:37:11 sean-k-mooney it sound like you have stale allocation in plamcent then
19:37:36 sean-k-mooney although what i dont understand is why the az filter removed any hosts
19:37:43 alexe9191 anyway to make sure that this is the case?
19:37:49 sean-k-mooney you said you enabeld the plamcent az filtering
19:38:03 alexe9191 no I meant on the api request
19:38:09 sean-k-mooney oh ok
19:38:21 alexe9191 apologies for the confusion:)
19:38:29 sean-k-mooney no worries
19:38:38 sean-k-mooney am we have a heal allcoation command
19:38:45 sean-k-mooney dansmith: do you know if that is in rocky
19:39:02 dansmith not off hand
19:40:12 sean-k-mooney so we have https://github.com/openstack/nova/blob/7b5ac717bd338be32414ae25f60a4bfe4c94c0f4/nova/cmd/manage.py#L2121
19:40:32 sean-k-mooney ya that is on rocky
19:41:14 sean-k-mooney so you can do nova-manage --heal-allocations i think but before you do that
19:41:33 sean-k-mooney alexe9191: you have the aggreate ram,disk,core filters enabled
19:41:42 alexe9191 indeed
19:41:45 sean-k-mooney alexe9191: do you manage allocation ratios by aggreate
19:42:05 alexe9191 more or less yes
19:42:32 sean-k-mooney ok that is proably the issue
19:42:41 sean-k-mooney http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html
19:43:00 sean-k-mooney we deprecated those in ocata because once you use placment you can nolonger do that
19:43:16 sean-k-mooney alexe9191: you have to set the allocation ratios per host
19:43:35 alexe9191 let me check the code I think we have that in place now
19:44:53 sean-k-mooney ok here are the docs on that topic if you have not got them set on each compute host
19:44:56 sean-k-mooney https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios
19:45:19 alexe9191 just checked right now and we have that on rocky
19:45:31 alexe9191 older versions are a different story but we are getting rid of those
19:45:35 sean-k-mooney ok so its in the nova.conf on each of your compute nodees
19:45:40 alexe9191 yes
19:46:26 alexe9191 cpu_allocation_ratio=1.0
19:46:26 alexe9191 docker exec -it nova_compute grep cpu /etc/nova/nova.conf
19:46:48 sean-k-mooney ok then what you basically need to do is compare the avaiable resouces in plamcnet to those reported in the hyperviors api
19:47:21 sean-k-mooney if there is a missmatch due to stales allcoation the usage wont be the same in nova view and placments
19:47:50 alexe9191 I am actually building now a json file with the resources using openstack hypervisor show
19:47:59 sean-k-mooney which would be why plamcent would have elminated the host before it got to the az filter
19:49:05 sean-k-mooney unfortunetly im more or less done for the day so i wont be able to help you contiue debuging but my best guess is placemetn and nova are out of sync
19:49:37 sean-k-mooney so placment is filtering out the host in that az first. before it gets to the schduler
19:50:00 alexe9191 You've already sat me on the good path:)  I am going to check this and check the heal command if that is the case
19:50:12 alexe9191 one more question though, can I query the resources per node? or is that per class only?
19:50:27 sean-k-mooney in placment?
19:50:31 alexe9191 yes
19:50:34 sean-k-mooney yes you can
19:50:53 sean-k-mooney you can use the resouce providers endpoint ot list all the invetores and the usage per host
19:51:10 sean-k-mooney there is an osc plugin for that too
19:51:27 sean-k-mooney called osc-placement if you dont have it installed
19:51:40 sean-k-mooney it will give you some placment command for the openstack client
19:52:16 alexe9191 installing now:)  thanks alot
19:53:00 sean-k-mooney here are the docs https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list
19:53:32 sean-k-mooney openstack resource provider show [--allocations] <uuid> will also be useful
19:53:54 sean-k-mooney or openstack resource provider usage show <uuid>
19:54:23 alexe9191 Testing it now :)
19:54:52 alexe9191 yup, got the numbers
20:10:52 alexe9191 yup, I see differences between what nova is reporting vs what placement is reporting
20:44:19 openstackgerrit Merged openstack/nova master: functional: Add tests for mixed CPU policy https://review.opendev.org/c/openstack/nova/+/755852
21:03:37 openstackgerrit Merged openstack/nova master: Remove __unicode__() from nova unit test Exception https://review.opendev.org/c/openstack/nova/+/769894
22:39:37 openstackgerrit Merged openstack/nova stable/ussuri: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/770764
22:40:19 openstack bug 1896463 in OpenStack Compute (nova) ussuri "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 - Assigned to Balazs Gibizer (balazs-gibizer)
22:40:19 openstackgerrit Merged openstack/nova stable/ussuri: Reproduce bug 1896463 in func env https://review.opendev.org/c/openstack/nova/+/770768
22:40:41 openstackgerrit Merged openstack/nova stable/ussuri: [doc]: Fix glance image_metadata link https://review.opendev.org/c/openstack/nova/+/761977
22:55:47 openstackgerrit Merged openstack/nova stable/train: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/767793
#openstack-nova - 2021-02-05
00:12:26 openstackgerrit Merged openstack/nova stable/train: Use cell targeted context to query BDMs for metadata https://review.opendev.org/c/openstack/nova/+/771407
00:28:54 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: libvirt: start tracking NUMACell.socket for hosts https://review.opendev.org/c/openstack/nova/+/766816
00:28:55 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: Add `socket` PCI NUMA affinity policy request prefilter https://review.opendev.org/c/openstack/nova/+/772749
00:28:55 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: extra specs/image pros: add `socket` PCI NUMA affinity https://review.opendev.org/c/openstack/nova/+/772748
00:28:56 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: pci: implement the `socket` NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779
00:28:56 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: Track host NUMA topology in PCI manager https://review.opendev.org/c/openstack/nova/+/774149
01:38:05 openstackgerrit Merged openstack/nova master: hyperv: Configures chassis asset tags for VMs. https://review.opendev.org/c/openstack/nova/+/752723
01:59:04 openstack bug 1907522 in OpenStack Compute (nova) ussuri "test_*_with_qos_min_bw_allocation fails in the nova-multi-cell job with: nova.exception.MigrationPreCheckError: Migration pre-check error: Failed to create port bindings for host " [Medium,In progress] https://launchpad.net/bugs/1907522 - Assigned to Balazs Gibizer (balazs-gibizer)
01:59:04 openstackgerrit Merged openstack/nova stable/ussuri: Reproduce bug 1907522 in functional test https://review.opendev.org/c/openstack/nova/+/773930
04:13:07 openstackgerrit Merged openstack/nova master: Fix ForbiddenWithAccelerators to HTTPForbidden for shelve API https://review.opendev.org/c/openstack/nova/+/770007
04:14:15 openstackgerrit Merged openstack/nova master: pci: Improve testing of 'nova.pci.request' module https://review.opendev.org/c/openstack/nova/+/764446
05:42:19 openstackgerrit Merged openstack/nova stable/ussuri: Set instance host and drop migration under lock https://review.opendev.org/c/openstack/nova/+/770769
06:01:07 openstackgerrit Khomesh Thakre proposed openstack/nova master: api: Reject requests to reset the state of SHELVE_OFFLOADED servers https://review.opendev.org/c/openstack/nova/+/773238
06:40:08 openstackgerrit Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509
06:40:09 openstackgerrit Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315
06:40:10 openstackgerrit Wenping Song proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852
08:18:07 openstackgerrit Wenping Song proposed openstack/nova master: Nova supports password encrypted VNC https://review.opendev.org/c/openstack/nova/+/622336
08:23:32 alexe9191 Good day everyone, is there an issue anyone aware of in OpenStack rocky in the scheduler? it seems to think that some of my hosts belong to the nova AZ, while nova-api is reporting that they are in a different zone.
08:49:42 alexe9191 somehow removing one host of the aggregate and re-adding it fixed the problem for the whole aggregate. I think there is some kind of a update all mechanism when adding/removing hosts to aggregates.
08:54:25 bauzas alexe9191: have you named an aggregate with a 'nova' name for its AZ ?
08:55:15 alexe9191 no
08:55:20 alexe9191 that's the default one
08:55:25 bauzas I mean, is there an aggregate having a metadata value for AZ being "nova" ?

Earlier   Later