| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-13 | |||
| 17:49:55 | sean-k-mooney | stable rocky is oslo.concurrency===3.27.0 | |
| 17:50:07 | TheJulia | markguz_: yeah, :\ | |
| 17:50:14 | melwitt | yes, the patch that added fair locks bumped the oslo.concurrency version | |
| 17:50:22 | sean-k-mooney | so it can go back to stien | |
| 17:50:32 | sean-k-mooney | but not rocky | |
| 17:50:40 | melwitt | so it wasn't bumped until ussuri | |
| 17:51:18 | sean-k-mooney | markguz_: ya you would need to know which host has it | |
| 17:51:33 | markguz_ | what's ironic is (pun intended) is that i was upgrading with the intention of getting to ussuri but when ironic failed at the rocky step i didn't want to compound the problem by continuing to upgrade | |
| 17:52:37 | melwitt | huh yeah actually it could be backported to stein because the upper constraint is 3.29.1 for whatever reason | |
| 17:52:56 | melwitt | I did not expect that | |
| 17:53:02 | markguz_ | assuming the bug is the problem going to ussuri will fix it? but that will break a lot of our automation due to the way were calling the nodes | |
| 17:53:25 | markguz_ | i mean it's not the end of the world, but ugh.. more work :-( | |
| 17:53:58 | markguz_ | at least i finally have a better idea of what's wrong at least. i seriously was losing the will to live over this ;-) | |
| 17:54:19 | melwitt | yeah. you could try to haxx and apply the patch to see if it helps. you just need oslo.concurrency >= 3.29.0 | |
| 17:54:45 | melwitt | (so you know for sure whether you're hitting that bug) | |
| 17:55:15 | markguz_ | melwitt: does the patch need to go on the scheduler or the compute node? or both? | |
| 17:55:25 | melwitt | markguz_: compute node | |
| 17:56:15 | markguz_ | melwitt: then i can probably crowbar that in | |
| 17:58:12 | melwitt | bleh, there's merge conflicts but it's really just adding fair=True to all the @utils.synchronized(COMPUTE_RESOURCE_SEMAPHORE, fair=True) | |
| 17:59:03 | markguz_ | ok. i'll give it try and see what happens. | |
| 18:00:03 | markguz_ | will i need to upgrade the other oslo. components or just concurrency? | |
| 18:00:40 | melwitt | just concurrency | |
| 18:02:03 | openstackgerrit | melanie witt proposed openstack/nova stable/train: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770585 | |
| 18:04:32 | sean-k-mooney | we could proably implenet a version of the patch for rocky too | |
| 18:04:40 | sean-k-mooney | that just did not use the fair lock form oslo | |
| 18:10:23 | sean-k-mooney | markguz_: this was the implemenation fo the fair lock https://github.com/openstack/oslo.concurrency/commit/2b55da68ae45ff45cba68672cdbc24342cf115f6 | |
| 18:13:32 | sean-k-mooney | markguz_: if you wanted to backport the upstream patch and then backport the implemantion of the fair lock into nova we could evaulate that or at least the stable team could | |
| 18:13:53 | sean-k-mooney | its just using https://fasteners.readthedocs.io/en/latest/api/lock.html#fasteners.lock.ReaderWriterLock | |
| 18:14:03 | sean-k-mooney | to actuly provide the fifo behavior | |
| 18:15:50 | sean-k-mooney | the version of fasteners on stable rocky has the required functionality | |
| 18:16:45 | sean-k-mooney | that said i think you can just bump the oslo.concurrancy version locally and locally apply the nova patch and it should run fine | |
| 18:19:52 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657 | |
| 18:20:40 | sean-k-mooney | stephenfin: by the way as far as i am aware we never us the cpu_toplopgy filed in the numa cell object to generate teh xml at all | |
| 18:20:47 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657 | |
| 18:21:58 | sean-k-mooney | the numa toplogy of the guest or host should have no impact on the cpu toplogy of the guest period | |
| 18:22:28 | sean-k-mooney | any other behviaor is inconsistent with the intended behviaor as discibed by the specs | |
| 18:58:19 | openstackgerrit | Merged openstack/nova stable/victoria: Omit resource inventories from placement update if zero https://review.opendev.org/c/openstack/nova/+/766177 | |
| 19:14:38 | markguz_ | sean-k-mooney: i think it would be simpler for me to just upgrade to ussuri | |
| 19:14:58 | sean-k-mooney | if that is an option yes | |
| 19:15:37 | openstackgerrit | Merged openstack/nova stable/victoria: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/761924 | |
| 19:15:43 | markguz_ | fortunately for me this is an internal deployment that is not used by paying customers so i have some degree of flexibility on it's availability | |
| 19:16:23 | sean-k-mooney | what do you use to deploy/manage it | |
| 19:21:11 | markguz_ | sean-k-mooney: originally i deployed kilo with rdo packstack. since then it's become a bit of a bit of hodgepodge of manual installs. I mostly use ansible to keep things up to date | |
| 19:21:48 | sean-k-mooney | ah i see | |
| 19:22:01 | sean-k-mooney | packstack has more or less been unsupported for a few years now | |
| 19:22:27 | sean-k-mooney | i think it still technicaly exists but redhat stop supporting in with our product in queens i think | |
| 19:22:47 | markguz_ | yeah. i generally just install from the package manager. and have some ansible plays that configure compute nodes etc etc. | |
| 19:22:49 | sean-k-mooney | e.g. we moved to require all customer deploy with triplo around queens | |
| 19:23:12 | markguz_ | i've got some bits an pieces that are installed from git master for things like magnum and heat. | |
| 19:23:13 | sean-k-mooney | markguz_: you should look into openstack ansible so | |
| 19:23:27 | markguz_ | it's on my todo list :-) | |
| 19:26:15 | markguz_ | The patch seems to have fixed my problem... spawning is happening at the expected rate now | |
| 19:26:47 | sean-k-mooney | did you just bump the oslo concurancy version and apply th nova patch | |
| 19:26:58 | markguz_ | sean-k-mooney: yup | |
| 19:27:12 | sean-k-mooney | there is still goign to be contention on the lock but it should now preserve order | |
| 19:27:35 | markguz_ | so spliting to multiple compute nodes is still probably the best path | |
| 19:30:15 | sean-k-mooney | long term proably but at leat your current issue is mitagated | |
| 19:30:29 | sean-k-mooney | i wont say solved but managemable | |
| 19:33:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: `socket` PCI NUMA-affinity Policy https://review.opendev.org/c/openstack/nova-specs/+/765551 | |
| 19:33:52 | artom | sean-k-mooney, stephenfin (though I suspect you're done for the day) ^^ | |
| 19:34:21 | sean-k-mooney | more or less ill leave it open however for tomorrow | |
| 19:34:35 | artom | ... with some snow shovelling and kid driving thrown in the mix | |
| 23:54:03 | openstackgerrit | Merged openstack/nova stable/stein: [stable-only] Cap bandit and make lower-constraints job non-voting https://review.opendev.org/c/openstack/nova/+/766487 | |
| #openstack-nova - 2021-01-14 | |||
| 00:41:48 | brinzhang_ | gmann: hi are you around? | |
| 00:42:08 | brinzhang_ | gmann: what's your mean of this https://review.opendev.org/c/openstack/nova/+/729563/26/nova/api/openstack/compute/shelve.py#59 | |
| 00:42:39 | brinzhang_ | gmann: do we need to merge your fix into the shelve/unshelve patch? | |
| 01:34:16 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 02:14:48 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 02:47:25 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 04:32:32 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851 | |
| 06:05:45 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Libvirt supports composing cyborg owned vGPU accelerator into domain XML https://review.opendev.org/c/openstack/nova-specs/+/750116 | |
| 06:08:48 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862 | |
| 06:08:49 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 08:39:52 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380 | |
| 08:39:53 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726 | |
| 09:06:48 | gibi | good morning | |
| 09:25:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/770764 | |
| 09:25:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/770765 | |
| 09:44:09 | lyarwood | sean-k-mooney: https://review.opendev.org/q/topic:spec/libvirt-default-machine-type - when you're around and have time can we have a chat about the direction of this series before I finish things up and write docs etc? | |
| 09:56:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Reproduce bug 1896463 in func env https://review.opendev.org/c/openstack/nova/+/770768 | |
| 09:56:37 | openstack | bug 1896463 in OpenStack Compute (nova) rocky "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 | |
| 09:56:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Set instance host and drop migration under lock https://review.opendev.org/c/openstack/nova/+/770769 | |
| 11:31:58 | sean-k-mooney | lyarwood: https://review.opendev.org/c/openstack/nova/+/767533/4/nova/virt/libvirt/driver.py#5775 | |
| 11:33:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: utils: Remove periods from instance hostnames https://review.opendev.org/c/openstack/nova/+/764482 | |
| 11:34:16 | sean-k-mooney | lyarwood: over all that pathch looks good to me be the precidence relationship is wrong in my view | |
| 11:34:45 | sean-k-mooney | lyarwood: im going to go grab coffee and ill be back in a minute but after that im free to talk whenever suits | |
| 11:41:41 | lyarwood | sean-k-mooney: cool thanks, image, sysmeta and then config. I'll respin this afternoon. | |
| 11:47:27 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/openstacksdk/+/767526 sorry about the spam on here btw, I'm using gibi's tampermonkey script to display zuul results and got confused when I saw failures from a previous run. | |
| 11:57:10 | stephenfin | lyarwood: heh, nw :) | |
| 12:08:16 | sean-k-mooney | stephenfin: https://review.opendev.org/c/openstack/python-openstackclient/+/765366 and the other osc changes faild the gate by the way | |
| 12:08:40 | sean-k-mooney | that one seams to have pagination issues for list servers which you did not change | |
| 12:08:51 | sean-k-mooney | it had a bunch of json decode errors | |
| 12:09:53 | sean-k-mooney | oh you have a recheck going | |
| 12:10:35 | sean-k-mooney | looks like the same error on all of them | |
| 12:35:08 | openstackgerrit | sean mooney proposed openstack/nova stable/victoria: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770744 | |
| 12:37:31 | openstackgerrit | sean mooney proposed openstack/nova stable/ussuri: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770745 | |
| 12:45:08 | openstackgerrit | sean mooney proposed openstack/nova stable/ussuri: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770745 | |
| 14:03:39 | gibi | lyarwood: I'm wondering if I should do the cleanup MIN_LIBVIRT_VERSION >= 4.1.0 before I change the detach logic as that cleanup would simplyfy the code that I have to handle | |
| 14:03:43 | gibi | https://github.com/openstack/nova/blob/e6f5e814050a19d6f027037424556b2889514ec3/nova/virt/libvirt/guest.py#L418 | |
| 14:16:25 | lyarwood | gibi: hey sorry, so that's all part of https://review.opendev.org/q/topic:%22bump-libvirt-qemu-wallaby%22 if I ever get around to it | |