Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-13
17:49:39 markguz_ if i run multiple computes i'm guessing that i will need to change how i call a instance. right now i use the "avail_zone:compute_host:bm_uuid" trick
17:49:40 melwitt yeah, I said all of that earlier
17:49:55 sean-k-mooney stable rocky is oslo.concurrency===3.27.0
17:50:07 TheJulia markguz_: yeah, :\
17:50:14 melwitt yes, the patch that added fair locks bumped the oslo.concurrency version
17:50:22 sean-k-mooney so it can go back to stien
17:50:32 sean-k-mooney but not rocky
17:50:40 melwitt so it wasn't bumped until ussuri
17:51:18 sean-k-mooney markguz_: ya you would need to know which host has it
17:51:33 markguz_ what's ironic is (pun intended) is that i was upgrading with the intention of getting to ussuri but when ironic failed at the rocky step i didn't want to compound the problem by continuing to upgrade
17:52:37 melwitt huh yeah actually it could be backported to stein because the upper constraint is 3.29.1 for whatever reason
17:52:56 melwitt I did not expect that
17:53:02 markguz_ assuming the bug is the problem going to ussuri will fix it? but that will break a lot of our automation due to the way were calling the nodes
17:53:25 markguz_ i mean it's not the end of the world, but ugh.. more work :-(
17:53:58 markguz_ at least i finally have a better idea of what's wrong at least. i seriously was losing the will to live over this ;-)
17:54:19 melwitt yeah. you could try to haxx and apply the patch to see if it helps. you just need oslo.concurrency >= 3.29.0
17:54:45 melwitt (so you know for sure whether you're hitting that bug)
17:55:15 markguz_ melwitt: does the patch need to go on the scheduler or the compute node? or both?
17:55:25 melwitt markguz_: compute node
17:56:15 markguz_ melwitt: then i can probably crowbar that in
17:58:12 melwitt bleh, there's merge conflicts but it's really just adding fair=True to all the @utils.synchronized(COMPUTE_RESOURCE_SEMAPHORE, fair=True)
17:59:03 markguz_ ok. i'll give it try and see what happens.
18:00:03 markguz_ will i need to upgrade the other oslo. components or just concurrency?
18:00:40 melwitt just concurrency
18:02:03 openstackgerrit melanie witt proposed openstack/nova stable/train: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770585
18:04:32 sean-k-mooney we could proably implenet a version of the patch for rocky too
18:04:40 sean-k-mooney that just did not use the fair lock form oslo
18:10:23 sean-k-mooney markguz_: this was the implemenation fo the fair lock https://github.com/openstack/oslo.concurrency/commit/2b55da68ae45ff45cba68672cdbc24342cf115f6
18:13:32 sean-k-mooney markguz_: if you wanted to backport the upstream patch and then backport the implemantion of the fair lock into nova we could evaulate that or at least the stable team could
18:13:53 sean-k-mooney its just using https://fasteners.readthedocs.io/en/latest/api/lock.html#fasteners.lock.ReaderWriterLock
18:14:03 sean-k-mooney to actuly provide the fifo behavior
18:15:50 sean-k-mooney the version of fasteners on stable rocky has the required functionality
18:16:45 sean-k-mooney that said i think you can just bump the oslo.concurrancy version locally and locally apply the nova patch and it should run fine
18:19:52 openstackgerrit melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657
18:20:40 sean-k-mooney stephenfin: by the way as far as i am aware we never us the cpu_toplopgy filed in the numa cell object to generate teh xml at all
18:20:47 openstackgerrit melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657
18:21:58 sean-k-mooney the numa toplogy of the guest or host should have no impact on the cpu toplogy of the guest period
18:22:28 sean-k-mooney any other behviaor is inconsistent with the intended behviaor as discibed by the specs
18:58:19 openstackgerrit Merged openstack/nova stable/victoria: Omit resource inventories from placement update if zero https://review.opendev.org/c/openstack/nova/+/766177
19:14:38 markguz_ sean-k-mooney: i think it would be simpler for me to just upgrade to ussuri
19:14:58 sean-k-mooney if that is an option yes
19:15:37 openstackgerrit Merged openstack/nova stable/victoria: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/761924
19:15:43 markguz_ fortunately for me this is an internal deployment that is not used by paying customers so i have some degree of flexibility on it's availability
19:16:23 sean-k-mooney what do you use to deploy/manage it
19:21:11 markguz_ sean-k-mooney: originally i deployed kilo with rdo packstack. since then it's become a bit of a bit of hodgepodge of manual installs. I mostly use ansible to keep things up to date
19:21:48 sean-k-mooney ah i see
19:22:01 sean-k-mooney packstack has more or less been unsupported for a few years now
19:22:27 sean-k-mooney i think it still technicaly exists but redhat stop supporting in with our product in queens i think
19:22:47 markguz_ yeah. i generally just install from the package manager. and have some ansible plays that configure compute nodes etc etc.
19:22:49 sean-k-mooney e.g. we moved to require all customer deploy with triplo around queens
19:23:12 markguz_ i've got some bits an pieces that are installed from git master for things like magnum and heat.
19:23:13 sean-k-mooney markguz_: you should look into openstack ansible so
19:23:27 markguz_ it's on my todo list :-)
19:26:15 markguz_ The patch seems to have fixed my problem... spawning is happening at the expected rate now
19:26:47 sean-k-mooney did you just bump the oslo concurancy version and apply th nova patch
19:26:58 markguz_ sean-k-mooney: yup
19:27:12 sean-k-mooney there is still goign to be contention on the lock but it should now preserve order
19:27:35 markguz_ so spliting to multiple compute nodes is still probably the best path
19:30:15 sean-k-mooney long term proably but at leat your current issue is mitagated
19:30:29 sean-k-mooney i wont say solved but managemable
19:33:33 openstackgerrit Artom Lifshitz proposed openstack/nova-specs master: `socket` PCI NUMA-affinity Policy https://review.opendev.org/c/openstack/nova-specs/+/765551
19:33:52 artom sean-k-mooney, stephenfin (though I suspect you're done for the day) ^^
19:34:21 sean-k-mooney more or less ill leave it open however for tomorrow
19:34:35 artom ... with some snow shovelling and kid driving thrown in the mix
23:54:03 openstackgerrit Merged openstack/nova stable/stein: [stable-only] Cap bandit and make lower-constraints job non-voting https://review.opendev.org/c/openstack/nova/+/766487
#openstack-nova - 2021-01-14
00:41:48 brinzhang_ gmann: hi are you around?
00:42:08 brinzhang_ gmann: what's your mean of this https://review.opendev.org/c/openstack/nova/+/729563/26/nova/api/openstack/compute/shelve.py#59
00:42:39 brinzhang_ gmann: do we need to merge your fix into the shelve/unshelve patch?
01:34:16 openstackgerrit Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509
02:14:48 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315
02:47:25 openstackgerrit Brin Zhang proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852
04:32:32 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851
06:05:45 openstackgerrit Wenping Song proposed openstack/nova-specs master: Libvirt supports composing cyborg owned vGPU accelerator into domain XML https://review.opendev.org/c/openstack/nova-specs/+/750116
06:08:48 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862
06:08:49 openstackgerrit Brin Zhang proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329
08:39:52 openstackgerrit Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380
08:39:53 openstackgerrit Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726
09:06:48 gibi good morning
09:25:30 openstackgerrit Balazs Gibizer proposed openstack/nova stable/ussuri: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/770764
09:25:31 openstackgerrit Balazs Gibizer proposed openstack/nova stable/ussuri: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/770765
09:44:09 lyarwood sean-k-mooney: https://review.opendev.org/q/topic:spec/libvirt-default-machine-type - when you're around and have time can we have a chat about the direction of this series before I finish things up and write docs etc?
09:56:36 openstackgerrit Balazs Gibizer proposed openstack/nova stable/ussuri: Reproduce bug 1896463 in func env https://review.opendev.org/c/openstack/nova/+/770768
09:56:37 openstack bug 1896463 in OpenStack Compute (nova) rocky "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463
09:56:38 openstackgerrit Balazs Gibizer proposed openstack/nova stable/ussuri: Set instance host and drop migration under lock https://review.opendev.org/c/openstack/nova/+/770769
11:31:58 sean-k-mooney lyarwood: https://review.opendev.org/c/openstack/nova/+/767533/4/nova/virt/libvirt/driver.py#5775
11:33:43 openstackgerrit Stephen Finucane proposed openstack/nova master: utils: Remove periods from instance hostnames https://review.opendev.org/c/openstack/nova/+/764482
11:34:16 sean-k-mooney lyarwood: over all that pathch looks good to me be the precidence relationship is wrong in my view
11:34:45 sean-k-mooney lyarwood: im going to go grab coffee and ill be back in a minute but after that im free to talk whenever suits
11:41:41 lyarwood sean-k-mooney: cool thanks, image, sysmeta and then config. I'll respin this afternoon.
11:47:27 lyarwood stephenfin: https://review.opendev.org/c/openstack/openstacksdk/+/767526 sorry about the spam on here btw, I'm using gibi's tampermonkey script to display zuul results and got confused when I saw failures from a previous run.
11:57:10 stephenfin lyarwood: heh, nw :)
12:08:16 sean-k-mooney stephenfin: https://review.opendev.org/c/openstack/python-openstackclient/+/765366 and the other osc changes faild the gate by the way
12:08:40 sean-k-mooney that one seams to have pagination issues for list servers which you did not change
12:08:51 sean-k-mooney it had a bunch of json decode errors
12:09:53 sean-k-mooney oh you have a recheck going
12:10:35 sean-k-mooney looks like the same error on all of them
12:35:08 openstackgerrit sean mooney proposed openstack/nova stable/victoria: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770744
12:37:31 openstackgerrit sean mooney proposed openstack/nova stable/ussuri: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770745
12:45:08 openstackgerrit sean mooney proposed openstack/nova stable/ussuri: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770745
14:03:39 gibi lyarwood: I'm wondering if I should do the cleanup MIN_LIBVIRT_VERSION >= 4.1.0 before I change the detach logic as that cleanup would simplyfy the code that I have to handle

Earlier   Later