Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-13
17:46:46 melwitt "conductor groups"
17:46:52 TheJulia melwitt: only to force specific grouping/allocation into specific grouping
17:46:56 sean-k-mooney that on the ironic side i think
17:47:01 melwitt it's not
17:47:05 TheJulia its on both sides
17:47:12 sean-k-mooney ah ok
17:47:19 markguz_ peridoc_task_interval is set to 240
17:47:19 melwitt well, it might be but you have to do it on the nova side too
17:47:24 TheJulia otherwise it runs a hash ring based upon the node list
17:47:31 TheJulia and the group is just a key in the hash ring
17:48:01 sean-k-mooney we improved this in nova by using oslos fair locks
17:48:02 TheJulia the nova side name is a little different because naming_is_fun^TM
17:48:02 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/711528/2
17:48:23 markguz_ so we use this in a lab env and when we spin up baremetal we need to spin up a spcific node as they are connected to specific hardware that is being tested
17:48:39 sean-k-mooney but that was only done in ussuri
17:48:48 TheJulia sean-k-mooney: ohhhhh neat
17:49:02 sean-k-mooney so we would have to backport it unfortunetlly im not sure oslo has the required support in rocky let me check
17:49:13 melwitt ok conductor groups are not available until stein anyways
17:49:30 sean-k-mooney we would need oslo.concurrancy 3.29.0 to backport it
17:49:39 markguz_ if i run multiple computes i'm guessing that i will need to change how i call a instance. right now i use the "avail_zone:compute_host:bm_uuid" trick
17:49:40 melwitt yeah, I said all of that earlier
17:49:55 sean-k-mooney stable rocky is oslo.concurrency===3.27.0
17:50:07 TheJulia markguz_: yeah, :\
17:50:14 melwitt yes, the patch that added fair locks bumped the oslo.concurrency version
17:50:22 sean-k-mooney so it can go back to stien
17:50:32 sean-k-mooney but not rocky
17:50:40 melwitt so it wasn't bumped until ussuri
17:51:18 sean-k-mooney markguz_: ya you would need to know which host has it
17:51:33 markguz_ what's ironic is (pun intended) is that i was upgrading with the intention of getting to ussuri but when ironic failed at the rocky step i didn't want to compound the problem by continuing to upgrade
17:52:37 melwitt huh yeah actually it could be backported to stein because the upper constraint is 3.29.1 for whatever reason
17:52:56 melwitt I did not expect that
17:53:02 markguz_ assuming the bug is the problem going to ussuri will fix it? but that will break a lot of our automation due to the way were calling the nodes
17:53:25 markguz_ i mean it's not the end of the world, but ugh.. more work :-(
17:53:58 markguz_ at least i finally have a better idea of what's wrong at least. i seriously was losing the will to live over this ;-)
17:54:19 melwitt yeah. you could try to haxx and apply the patch to see if it helps. you just need oslo.concurrency >= 3.29.0
17:54:45 melwitt (so you know for sure whether you're hitting that bug)
17:55:15 markguz_ melwitt: does the patch need to go on the scheduler or the compute node? or both?
17:55:25 melwitt markguz_: compute node
17:56:15 markguz_ melwitt: then i can probably crowbar that in
17:58:12 melwitt bleh, there's merge conflicts but it's really just adding fair=True to all the @utils.synchronized(COMPUTE_RESOURCE_SEMAPHORE, fair=True)
17:59:03 markguz_ ok. i'll give it try and see what happens.
18:00:03 markguz_ will i need to upgrade the other oslo. components or just concurrency?
18:00:40 melwitt just concurrency
18:02:03 openstackgerrit melanie witt proposed openstack/nova stable/train: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770585
18:04:32 sean-k-mooney we could proably implenet a version of the patch for rocky too
18:04:40 sean-k-mooney that just did not use the fair lock form oslo
18:10:23 sean-k-mooney markguz_: this was the implemenation fo the fair lock https://github.com/openstack/oslo.concurrency/commit/2b55da68ae45ff45cba68672cdbc24342cf115f6
18:13:32 sean-k-mooney markguz_: if you wanted to backport the upstream patch and then backport the implemantion of the fair lock into nova we could evaulate that or at least the stable team could
18:13:53 sean-k-mooney its just using https://fasteners.readthedocs.io/en/latest/api/lock.html#fasteners.lock.ReaderWriterLock
18:14:03 sean-k-mooney to actuly provide the fifo behavior
18:15:50 sean-k-mooney the version of fasteners on stable rocky has the required functionality
18:16:45 sean-k-mooney that said i think you can just bump the oslo.concurrancy version locally and locally apply the nova patch and it should run fine
18:19:52 openstackgerrit melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657
18:20:40 sean-k-mooney stephenfin: by the way as far as i am aware we never us the cpu_toplopgy filed in the numa cell object to generate teh xml at all
18:20:47 openstackgerrit melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657
18:21:58 sean-k-mooney the numa toplogy of the guest or host should have no impact on the cpu toplogy of the guest period
18:22:28 sean-k-mooney any other behviaor is inconsistent with the intended behviaor as discibed by the specs
18:58:19 openstackgerrit Merged openstack/nova stable/victoria: Omit resource inventories from placement update if zero https://review.opendev.org/c/openstack/nova/+/766177
19:14:38 markguz_ sean-k-mooney: i think it would be simpler for me to just upgrade to ussuri
19:14:58 sean-k-mooney if that is an option yes
19:15:37 openstackgerrit Merged openstack/nova stable/victoria: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/761924
19:15:43 markguz_ fortunately for me this is an internal deployment that is not used by paying customers so i have some degree of flexibility on it's availability
19:16:23 sean-k-mooney what do you use to deploy/manage it
19:21:11 markguz_ sean-k-mooney: originally i deployed kilo with rdo packstack. since then it's become a bit of a bit of hodgepodge of manual installs. I mostly use ansible to keep things up to date
19:21:48 sean-k-mooney ah i see
19:22:01 sean-k-mooney packstack has more or less been unsupported for a few years now
19:22:27 sean-k-mooney i think it still technicaly exists but redhat stop supporting in with our product in queens i think
19:22:47 markguz_ yeah. i generally just install from the package manager. and have some ansible plays that configure compute nodes etc etc.
19:22:49 sean-k-mooney e.g. we moved to require all customer deploy with triplo around queens
19:23:12 markguz_ i've got some bits an pieces that are installed from git master for things like magnum and heat.
19:23:13 sean-k-mooney markguz_: you should look into openstack ansible so
19:23:27 markguz_ it's on my todo list :-)
19:26:15 markguz_ The patch seems to have fixed my problem... spawning is happening at the expected rate now
19:26:47 sean-k-mooney did you just bump the oslo concurancy version and apply th nova patch
19:26:58 markguz_ sean-k-mooney: yup
19:27:12 sean-k-mooney there is still goign to be contention on the lock but it should now preserve order
19:27:35 markguz_ so spliting to multiple compute nodes is still probably the best path
19:30:15 sean-k-mooney long term proably but at leat your current issue is mitagated
19:30:29 sean-k-mooney i wont say solved but managemable
19:33:33 openstackgerrit Artom Lifshitz proposed openstack/nova-specs master: `socket` PCI NUMA-affinity Policy https://review.opendev.org/c/openstack/nova-specs/+/765551
19:33:52 artom sean-k-mooney, stephenfin (though I suspect you're done for the day) ^^
19:34:21 sean-k-mooney more or less ill leave it open however for tomorrow
19:34:35 artom ... with some snow shovelling and kid driving thrown in the mix
23:54:03 openstackgerrit Merged openstack/nova stable/stein: [stable-only] Cap bandit and make lower-constraints job non-voting https://review.opendev.org/c/openstack/nova/+/766487
#openstack-nova - 2021-01-14
00:41:48 brinzhang_ gmann: hi are you around?
00:42:08 brinzhang_ gmann: what's your mean of this https://review.opendev.org/c/openstack/nova/+/729563/26/nova/api/openstack/compute/shelve.py#59
00:42:39 brinzhang_ gmann: do we need to merge your fix into the shelve/unshelve patch?
01:34:16 openstackgerrit Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509
02:14:48 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315
02:47:25 openstackgerrit Brin Zhang proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852
04:32:32 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851
06:05:45 openstackgerrit Wenping Song proposed openstack/nova-specs master: Libvirt supports composing cyborg owned vGPU accelerator into domain XML https://review.opendev.org/c/openstack/nova-specs/+/750116
06:08:48 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862
06:08:49 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329
08:39:52 openstackgerrit Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380
08:39:53 openstackgerrit Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726
09:06:48 gibi good morning
09:25:30 openstackgerrit Balazs Gibizer proposed openstack/nova stable/ussuri: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/770764
09:25:31 openstackgerrit Balazs Gibizer proposed openstack/nova stable/ussuri: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/770765
09:44:09 lyarwood sean-k-mooney: https://review.opendev.org/q/topic:spec/libvirt-default-machine-type - when you're around and have time can we have a chat about the direction of this series before I finish things up and write docs etc?

Earlier   Later