| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-04 | |||
| 19:31:32 | alexe9191 | actually what I said was just wrong.. i end up with the same number of hosts 148 if I drop the az, the scheduling happen though cause az filter is not filtering anything out | |
| 19:31:41 | alexe9191 | yes one moment let me sanitise it | |
| 19:32:13 | sean-k-mooney | ya if you have enable the plamcent version you can also disable the az filter | |
| 19:32:31 | alexe9191 | indeed that will also be done since it can be used | |
| 19:32:36 | alexe9191 | so here is the version that works: | |
| 19:32:36 | alexe9191 | http://paste.openstack.org/show/802341/ | |
| 19:32:37 | sean-k-mooney | over the release we have slowly been moving things too placment where it makes sense | |
| 19:33:15 | sean-k-mooney | yep so you went form 148 down to 104 | |
| 19:33:25 | sean-k-mooney | then those would get weighed | |
| 19:33:58 | alexe9191 | http://paste.openstack.org/show/802342/ this is the one that does not (COmpute is reporting 0 cause those are disabled) | |
| 19:34:20 | alexe9191 | Filter AvailabilityZoneFilter returned 8 because 8 are only in zone1 | |
| 19:34:28 | alexe9191 | so I am starting with less than I should | |
| 19:34:57 | sean-k-mooney | so the first following filters RetryFilter AvailabilityZoneFilter AggregateDiskFilter, AggregateCoreFilter and AggregateRamFilter can be removed | |
| 19:35:07 | sean-k-mooney | and NUMATopologyFilter should come last | |
| 19:36:07 | sean-k-mooney | so the one that faild had most of the host elimiated by the AvailabilityZoneFilter | |
| 19:36:29 | alexe9191 | indeed, because none of the hosts in zone1 reported it self to placement to be a good match for that flavor | |
| 19:36:40 | alexe9191 | though there are plenty of space on those hosts to cover the needed resources | |
| 19:36:47 | alexe9191 | and I have no max placement in the config | |
| 19:37:11 | sean-k-mooney | it sound like you have stale allocation in plamcent then | |
| 19:37:36 | sean-k-mooney | although what i dont understand is why the az filter removed any hosts | |
| 19:37:43 | alexe9191 | anyway to make sure that this is the case? | |
| 19:37:49 | sean-k-mooney | you said you enabeld the plamcent az filtering | |
| 19:38:03 | alexe9191 | no I meant on the api request | |
| 19:38:09 | sean-k-mooney | oh ok | |
| 19:38:21 | alexe9191 | apologies for the confusion:) | |
| 19:38:29 | sean-k-mooney | no worries | |
| 19:38:38 | sean-k-mooney | am we have a heal allcoation command | |
| 19:38:45 | sean-k-mooney | dansmith: do you know if that is in rocky | |
| 19:39:02 | dansmith | not off hand | |
| 19:40:12 | sean-k-mooney | so we have https://github.com/openstack/nova/blob/7b5ac717bd338be32414ae25f60a4bfe4c94c0f4/nova/cmd/manage.py#L2121 | |
| 19:40:32 | sean-k-mooney | ya that is on rocky | |
| 19:41:14 | sean-k-mooney | so you can do nova-manage --heal-allocations i think but before you do that | |
| 19:41:33 | sean-k-mooney | alexe9191: you have the aggreate ram,disk,core filters enabled | |
| 19:41:42 | alexe9191 | indeed | |
| 19:41:45 | sean-k-mooney | alexe9191: do you manage allocation ratios by aggreate | |
| 19:42:05 | alexe9191 | more or less yes | |
| 19:42:32 | sean-k-mooney | ok that is proably the issue | |
| 19:42:41 | sean-k-mooney | http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html | |
| 19:43:00 | sean-k-mooney | we deprecated those in ocata because once you use placment you can nolonger do that | |
| 19:43:16 | sean-k-mooney | alexe9191: you have to set the allocation ratios per host | |
| 19:43:35 | alexe9191 | let me check the code I think we have that in place now | |
| 19:44:53 | sean-k-mooney | ok here are the docs on that topic if you have not got them set on each compute host | |
| 19:44:56 | sean-k-mooney | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios | |
| 19:45:19 | alexe9191 | just checked right now and we have that on rocky | |
| 19:45:31 | alexe9191 | older versions are a different story but we are getting rid of those | |
| 19:45:35 | sean-k-mooney | ok so its in the nova.conf on each of your compute nodees | |
| 19:45:40 | alexe9191 | yes | |
| 19:46:26 | alexe9191 | docker exec -it nova_compute grep cpu /etc/nova/nova.conf | |
| 19:46:26 | alexe9191 | cpu_allocation_ratio=1.0 | |
| 19:46:48 | sean-k-mooney | ok then what you basically need to do is compare the avaiable resouces in plamcnet to those reported in the hyperviors api | |
| 19:47:21 | sean-k-mooney | if there is a missmatch due to stales allcoation the usage wont be the same in nova view and placments | |
| 19:47:50 | alexe9191 | I am actually building now a json file with the resources using openstack hypervisor show | |
| 19:47:59 | sean-k-mooney | which would be why plamcent would have elminated the host before it got to the az filter | |
| 19:49:05 | sean-k-mooney | unfortunetly im more or less done for the day so i wont be able to help you contiue debuging but my best guess is placemetn and nova are out of sync | |
| 19:49:37 | sean-k-mooney | so placment is filtering out the host in that az first. before it gets to the schduler | |
| 19:50:00 | alexe9191 | You've already sat me on the good path:) I am going to check this and check the heal command if that is the case | |
| 19:50:12 | alexe9191 | one more question though, can I query the resources per node? or is that per class only? | |
| 19:50:27 | sean-k-mooney | in placment? | |
| 19:50:31 | alexe9191 | yes | |
| 19:50:34 | sean-k-mooney | yes you can | |
| 19:50:53 | sean-k-mooney | you can use the resouce providers endpoint ot list all the invetores and the usage per host | |
| 19:51:10 | sean-k-mooney | there is an osc plugin for that too | |
| 19:51:27 | sean-k-mooney | called osc-placement if you dont have it installed | |
| 19:51:40 | sean-k-mooney | it will give you some placment command for the openstack client | |
| 19:52:16 | alexe9191 | installing now:) thanks alot | |
| 19:53:00 | sean-k-mooney | here are the docs https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list | |
| 19:53:32 | sean-k-mooney | openstack resource provider show [--allocations] <uuid> will also be useful | |
| 19:53:54 | sean-k-mooney | or openstack resource provider usage show <uuid> | |
| 19:54:23 | alexe9191 | Testing it now :) | |
| 19:54:52 | alexe9191 | yup, got the numbers | |
| 20:10:52 | alexe9191 | yup, I see differences between what nova is reporting vs what placement is reporting | |
| 20:44:19 | openstackgerrit | Merged openstack/nova master: functional: Add tests for mixed CPU policy https://review.opendev.org/c/openstack/nova/+/755852 | |
| 21:03:37 | openstackgerrit | Merged openstack/nova master: Remove __unicode__() from nova unit test Exception https://review.opendev.org/c/openstack/nova/+/769894 | |
| 22:39:37 | openstackgerrit | Merged openstack/nova stable/ussuri: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/770764 | |
| 22:40:19 | openstackgerrit | Merged openstack/nova stable/ussuri: Reproduce bug 1896463 in func env https://review.opendev.org/c/openstack/nova/+/770768 | |
| 22:40:19 | openstack | bug 1896463 in OpenStack Compute (nova) ussuri "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 22:40:41 | openstackgerrit | Merged openstack/nova stable/ussuri: [doc]: Fix glance image_metadata link https://review.opendev.org/c/openstack/nova/+/761977 | |
| 22:55:47 | openstackgerrit | Merged openstack/nova stable/train: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/767793 | |
| #openstack-nova - 2021-02-05 | |||
| 00:12:26 | openstackgerrit | Merged openstack/nova stable/train: Use cell targeted context to query BDMs for metadata https://review.opendev.org/c/openstack/nova/+/771407 | |
| 00:28:54 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: libvirt: start tracking NUMACell.socket for hosts https://review.opendev.org/c/openstack/nova/+/766816 | |
| 00:28:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: extra specs/image pros: add `socket` PCI NUMA affinity https://review.opendev.org/c/openstack/nova/+/772748 | |
| 00:28:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Add `socket` PCI NUMA affinity policy request prefilter https://review.opendev.org/c/openstack/nova/+/772749 | |
| 00:28:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Track host NUMA topology in PCI manager https://review.opendev.org/c/openstack/nova/+/774149 | |
| 00:28:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: pci: implement the `socket` NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779 | |
| 01:38:05 | openstackgerrit | Merged openstack/nova master: hyperv: Configures chassis asset tags for VMs. https://review.opendev.org/c/openstack/nova/+/752723 | |
| 01:59:04 | openstackgerrit | Merged openstack/nova stable/ussuri: Reproduce bug 1907522 in functional test https://review.opendev.org/c/openstack/nova/+/773930 | |
| 01:59:04 | openstack | bug 1907522 in OpenStack Compute (nova) ussuri "test_*_with_qos_min_bw_allocation fails in the nova-multi-cell job with: nova.exception.MigrationPreCheckError: Migration pre-check error: Failed to create port bindings for host |
|
| 04:13:07 | openstackgerrit | Merged openstack/nova master: Fix ForbiddenWithAccelerators to HTTPForbidden for shelve API https://review.opendev.org/c/openstack/nova/+/770007 | |
| 04:14:15 | openstackgerrit | Merged openstack/nova master: pci: Improve testing of 'nova.pci.request' module https://review.opendev.org/c/openstack/nova/+/764446 | |
| 05:42:19 | openstackgerrit | Merged openstack/nova stable/ussuri: Set instance host and drop migration under lock https://review.opendev.org/c/openstack/nova/+/770769 | |
| 06:01:07 | openstackgerrit | Khomesh Thakre proposed openstack/nova master: api: Reject requests to reset the state of SHELVE_OFFLOADED servers https://review.opendev.org/c/openstack/nova/+/773238 | |
| 06:40:08 | openstackgerrit | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 06:40:09 | openstackgerrit | Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 06:40:10 | openstackgerrit | Wenping Song proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 08:18:07 | openstackgerrit | Wenping Song proposed openstack/nova master: Nova supports password encrypted VNC https://review.opendev.org/c/openstack/nova/+/622336 | |
| 08:23:32 | alexe9191 | Good day everyone, is there an issue anyone aware of in OpenStack rocky in the scheduler? it seems to think that some of my hosts belong to the nova AZ, while nova-api is reporting that they are in a different zone. | |
| 08:49:42 | alexe9191 | somehow removing one host of the aggregate and re-adding it fixed the problem for the whole aggregate. I think there is some kind of a update all mechanism when adding/removing hosts to aggregates. | |
| 08:54:25 | bauzas | alexe9191: have you named an aggregate with a 'nova' name for its AZ ? | |
| 08:55:15 | alexe9191 | no | |
| 08:55:20 | alexe9191 | that's the default one | |