| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-13 | |||
| 17:51:33 | markguz_ | what's ironic is (pun intended) is that i was upgrading with the intention of getting to ussuri but when ironic failed at the rocky step i didn't want to compound the problem by continuing to upgrade | |
| 17:52:37 | melwitt | huh yeah actually it could be backported to stein because the upper constraint is 3.29.1 for whatever reason | |
| 17:52:56 | melwitt | I did not expect that | |
| 17:53:02 | markguz_ | assuming the bug is the problem going to ussuri will fix it? but that will break a lot of our automation due to the way were calling the nodes | |
| 17:53:25 | markguz_ | i mean it's not the end of the world, but ugh.. more work :-( | |
| 17:53:58 | markguz_ | at least i finally have a better idea of what's wrong at least. i seriously was losing the will to live over this ;-) | |
| 17:54:19 | melwitt | yeah. you could try to haxx and apply the patch to see if it helps. you just need oslo.concurrency >= 3.29.0 | |
| 17:54:45 | melwitt | (so you know for sure whether you're hitting that bug) | |
| 17:55:15 | markguz_ | melwitt: does the patch need to go on the scheduler or the compute node? or both? | |
| 17:55:25 | melwitt | markguz_: compute node | |
| 17:56:15 | markguz_ | melwitt: then i can probably crowbar that in | |
| 17:58:12 | melwitt | bleh, there's merge conflicts but it's really just adding fair=True to all the @utils.synchronized(COMPUTE_RESOURCE_SEMAPHORE, fair=True) | |
| 17:59:03 | markguz_ | ok. i'll give it try and see what happens. | |
| 18:00:03 | markguz_ | will i need to upgrade the other oslo. components or just concurrency? | |
| 18:00:40 | melwitt | just concurrency | |
| 18:02:03 | openstackgerrit | melanie witt proposed openstack/nova stable/train: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770585 | |
| 18:04:32 | sean-k-mooney | we could proably implenet a version of the patch for rocky too | |
| 18:04:40 | sean-k-mooney | that just did not use the fair lock form oslo | |
| 18:10:23 | sean-k-mooney | markguz_: this was the implemenation fo the fair lock https://github.com/openstack/oslo.concurrency/commit/2b55da68ae45ff45cba68672cdbc24342cf115f6 | |
| 18:13:32 | sean-k-mooney | markguz_: if you wanted to backport the upstream patch and then backport the implemantion of the fair lock into nova we could evaulate that or at least the stable team could | |
| 18:13:53 | sean-k-mooney | its just using https://fasteners.readthedocs.io/en/latest/api/lock.html#fasteners.lock.ReaderWriterLock | |
| 18:14:03 | sean-k-mooney | to actuly provide the fifo behavior | |
| 18:15:50 | sean-k-mooney | the version of fasteners on stable rocky has the required functionality | |
| 18:16:45 | sean-k-mooney | that said i think you can just bump the oslo.concurrancy version locally and locally apply the nova patch and it should run fine | |
| 18:19:52 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657 | |
| 18:20:40 | sean-k-mooney | stephenfin: by the way as far as i am aware we never us the cpu_toplopgy filed in the numa cell object to generate teh xml at all | |
| 18:20:47 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Use fair locks in resource tracker https://review.opendev.org/c/openstack/nova/+/770657 | |
| 18:21:58 | sean-k-mooney | the numa toplogy of the guest or host should have no impact on the cpu toplogy of the guest period | |
| 18:22:28 | sean-k-mooney | any other behviaor is inconsistent with the intended behviaor as discibed by the specs | |
| 18:58:19 | openstackgerrit | Merged openstack/nova stable/victoria: Omit resource inventories from placement update if zero https://review.opendev.org/c/openstack/nova/+/766177 | |
| 19:14:38 | markguz_ | sean-k-mooney: i think it would be simpler for me to just upgrade to ussuri | |
| 19:14:58 | sean-k-mooney | if that is an option yes | |
| 19:15:37 | openstackgerrit | Merged openstack/nova stable/victoria: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/761924 | |
| 19:15:43 | markguz_ | fortunately for me this is an internal deployment that is not used by paying customers so i have some degree of flexibility on it's availability | |
| 19:16:23 | sean-k-mooney | what do you use to deploy/manage it | |
| 19:21:11 | markguz_ | sean-k-mooney: originally i deployed kilo with rdo packstack. since then it's become a bit of a bit of hodgepodge of manual installs. I mostly use ansible to keep things up to date | |
| 19:21:48 | sean-k-mooney | ah i see | |
| 19:22:01 | sean-k-mooney | packstack has more or less been unsupported for a few years now | |
| 19:22:27 | sean-k-mooney | i think it still technicaly exists but redhat stop supporting in with our product in queens i think | |
| 19:22:47 | markguz_ | yeah. i generally just install from the package manager. and have some ansible plays that configure compute nodes etc etc. | |
| 19:22:49 | sean-k-mooney | e.g. we moved to require all customer deploy with triplo around queens | |
| 19:23:12 | markguz_ | i've got some bits an pieces that are installed from git master for things like magnum and heat. | |
| 19:23:13 | sean-k-mooney | markguz_: you should look into openstack ansible so | |
| 19:23:27 | markguz_ | it's on my todo list :-) | |
| 19:26:15 | markguz_ | The patch seems to have fixed my problem... spawning is happening at the expected rate now | |
| 19:26:47 | sean-k-mooney | did you just bump the oslo concurancy version and apply th nova patch | |
| 19:26:58 | markguz_ | sean-k-mooney: yup | |
| 19:27:12 | sean-k-mooney | there is still goign to be contention on the lock but it should now preserve order | |
| 19:27:35 | markguz_ | so spliting to multiple compute nodes is still probably the best path | |
| 19:30:15 | sean-k-mooney | long term proably but at leat your current issue is mitagated | |
| 19:30:29 | sean-k-mooney | i wont say solved but managemable | |
| 19:33:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: `socket` PCI NUMA-affinity Policy https://review.opendev.org/c/openstack/nova-specs/+/765551 | |
| 19:33:52 | artom | sean-k-mooney, stephenfin (though I suspect you're done for the day) ^^ | |
| 19:34:21 | sean-k-mooney | more or less ill leave it open however for tomorrow | |
| 19:34:35 | artom | ... with some snow shovelling and kid driving thrown in the mix | |
| 23:54:03 | openstackgerrit | Merged openstack/nova stable/stein: [stable-only] Cap bandit and make lower-constraints job non-voting https://review.opendev.org/c/openstack/nova/+/766487 | |
| #openstack-nova - 2021-01-14 | |||
| 00:41:48 | brinzhang_ | gmann: hi are you around? | |
| 00:42:08 | brinzhang_ | gmann: what's your mean of this https://review.opendev.org/c/openstack/nova/+/729563/26/nova/api/openstack/compute/shelve.py#59 | |
| 00:42:39 | brinzhang_ | gmann: do we need to merge your fix into the shelve/unshelve patch? | |
| 01:34:16 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 02:14:48 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 02:47:25 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage https://review.opendev.org/c/openstack/nova/+/768852 | |
| 04:32:32 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in os-quota-sets path https://review.opendev.org/c/openstack/nova/+/768851 | |
| 06:05:45 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Libvirt supports composing cyborg owned vGPU accelerator into domain XML https://review.opendev.org/c/openstack/nova-specs/+/750116 | |
| 06:08:48 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenant_id with project_id in Limits API https://review.opendev.org/c/openstack/nova/+/768862 | |
| 06:08:49 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replace tenant* with project* in codes https://review.opendev.org/c/openstack/nova/+/769329 | |
| 08:39:52 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from Rebuild Server API https://review.opendev.org/c/openstack/nova/+/766380 | |
| 08:39:53 | openstackgerrit | Brin Zhang proposed openstack/nova master: Replaces tenant_id with project_id from List SG API https://review.opendev.org/c/openstack/nova/+/766726 | |
| 09:06:48 | gibi | good morning | |
| 09:25:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/770764 | |
| 09:25:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/770765 | |
| 09:44:09 | lyarwood | sean-k-mooney: https://review.opendev.org/q/topic:spec/libvirt-default-machine-type - when you're around and have time can we have a chat about the direction of this series before I finish things up and write docs etc? | |
| 09:56:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Reproduce bug 1896463 in func env https://review.opendev.org/c/openstack/nova/+/770768 | |
| 09:56:37 | openstack | bug 1896463 in OpenStack Compute (nova) rocky "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,In progress] https://launchpad.net/bugs/1896463 | |
| 09:56:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/ussuri: Set instance host and drop migration under lock https://review.opendev.org/c/openstack/nova/+/770769 | |
| 11:31:58 | sean-k-mooney | lyarwood: https://review.opendev.org/c/openstack/nova/+/767533/4/nova/virt/libvirt/driver.py#5775 | |
| 11:33:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: utils: Remove periods from instance hostnames https://review.opendev.org/c/openstack/nova/+/764482 | |
| 11:34:16 | sean-k-mooney | lyarwood: over all that pathch looks good to me be the precidence relationship is wrong in my view | |
| 11:34:45 | sean-k-mooney | lyarwood: im going to go grab coffee and ill be back in a minute but after that im free to talk whenever suits | |
| 11:41:41 | lyarwood | sean-k-mooney: cool thanks, image, sysmeta and then config. I'll respin this afternoon. | |
| 11:47:27 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/openstacksdk/+/767526 sorry about the spam on here btw, I'm using gibi's tampermonkey script to display zuul results and got confused when I saw failures from a previous run. | |
| 11:57:10 | stephenfin | lyarwood: heh, nw :) | |
| 12:08:16 | sean-k-mooney | stephenfin: https://review.opendev.org/c/openstack/python-openstackclient/+/765366 and the other osc changes faild the gate by the way | |
| 12:08:40 | sean-k-mooney | that one seams to have pagination issues for list servers which you did not change | |
| 12:08:51 | sean-k-mooney | it had a bunch of json decode errors | |
| 12:09:53 | sean-k-mooney | oh you have a recheck going | |
| 12:10:35 | sean-k-mooney | looks like the same error on all of them | |
| 12:35:08 | openstackgerrit | sean mooney proposed openstack/nova stable/victoria: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770744 | |
| 12:37:31 | openstackgerrit | sean mooney proposed openstack/nova stable/ussuri: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770745 | |
| 12:45:08 | openstackgerrit | sean mooney proposed openstack/nova stable/ussuri: only wait for plugtime events in pre-live-migration https://review.opendev.org/c/openstack/nova/+/770745 | |
| 14:03:39 | gibi | lyarwood: I'm wondering if I should do the cleanup MIN_LIBVIRT_VERSION >= 4.1.0 before I change the detach logic as that cleanup would simplyfy the code that I have to handle | |
| 14:03:43 | gibi | https://github.com/openstack/nova/blob/e6f5e814050a19d6f027037424556b2889514ec3/nova/virt/libvirt/guest.py#L418 | |
| 14:16:25 | lyarwood | gibi: hey sorry, so that's all part of https://review.opendev.org/q/topic:%22bump-libvirt-qemu-wallaby%22 if I ever get around to it | |
| 14:16:32 | lyarwood | gibi: which actually came up downstream earlier today | |
| 14:16:50 | lyarwood | gibi: I can refresh that and try to remove that logic for you later today if you'd like | |
| 14:16:55 | lyarwood | gibi: avoid adding another thing to your plate | |
| 14:16:57 | stephenfin | sean-k-mooney: Sorry, missed the ping. Yeah, the functional tests for OSC aren't great. They're not fully isolated which means running them in parallel can produce all kinds of weird results, depending on the speed on the CI node | |
| 14:17:19 | sean-k-mooney | ah i see | |
| 14:17:20 | stephenfin | sean-k-mooney: so rechecking is the only thing to do until I've time to rework them all | |
| 14:17:53 | sean-k-mooney | well that or run the serially but that proably will be too long | |