| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-17 | |||
| 14:39:22 | openstackgerrit | YumengBao proposed openstack/nova-specs master: libvirt supports composing cyborg owned vGPU accelerator into domain XML https://review.opendev.org/c/openstack/nova-specs/+/750116 | |
| 14:40:30 | stephenfin | bauzas: any chance you could look at https://review.opendev.org/q/topic:%2522sriov-functional-tests%2522+status:open too | |
| 14:40:31 | stephenfin | ?? | |
| 14:45:16 | Yumeng | gibi: just updated vGPU spec according to today's discuss. Please help to mention in nova's meeting today so cores can know about the update! need their review and comment,Thanks! https://review.opendev.org/c/openstack/nova-specs/+/750116 | |
| 14:45:59 | gibi | Yumeng: thanks for the update. I will mention the fact that the spec needs core review on today's meeting | |
| 14:52:19 | openstackgerrit | Lance Bragstad proposed openstack/placement master: Bump oslo.log version to 4.3.0 https://review.opendev.org/c/openstack/placement/+/760229 | |
| 14:53:54 | openstackgerrit | Lance Bragstad proposed openstack/placement master: Pass context objects to oslo.policy directly https://review.opendev.org/c/openstack/placement/+/766785 | |
| 14:54:08 | openstackgerrit | Aleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout https://review.opendev.org/c/openstack/nova/+/764927 | |
| 15:02:33 | openstackgerrit | Lance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers https://review.opendev.org/c/openstack/placement/+/760240 | |
| 15:02:56 | openstackgerrit | Aleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout https://review.opendev.org/c/openstack/nova/+/764927 | |
| 15:13:34 | bauzas | Yumeng: gibi: saw the update but honestly, I won't have time for reviewing this large spec before end of this year | |
| 15:13:57 | bauzas | which is for me in 24 hours (with Friday afternoon full of meetings) | |
| 15:17:53 | gibi | bauzas: no worries | |
| 15:18:12 | gibi | Yumeng knows that we are just about to go for PTO for two weeks | |
| 15:23:42 | bauzas | gibi: actually, I lied, I just commented out loud | |
| 15:24:06 | bauzas | because I was horrified by what I saw | |
| 15:35:11 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: WIP run-evacuate-hook: Use osc to evacuate instances https://review.opendev.org/c/openstack/nova/+/767570 | |
| 15:36:12 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 15:36:12 | gibi | sean-k-mooney: you were right and I was wrong. https://bugs.launchpad.net/nova/+bug/1823251 is a timeout issue really. Now I can reporduce it locally. | |
| 15:36:42 | sean-k-mooney | gibi: oh ok so can we do what manial did or is it different? | |
| 15:37:08 | gibi | sean-k-mooney: I think we can try what manial did, but I'm affraid at some point we will run out the time budget for the whole test run | |
| 15:37:17 | gibi | test job run | |
| 15:37:38 | sean-k-mooney | you mentioned something about scale factor in the bug | |
| 15:37:53 | gibi | yeah, we have a 160sec per test case timeout | |
| 15:37:54 | sean-k-mooney | ah self.useFixture(timeout.Timeout( | |
| 15:37:55 | sean-k-mooney | scaling_factor=self.TIMEOUT_SCALING_FACTOR)) | |
| 15:38:08 | gibi | but the mysql tests uses 4 * 160 sec | |
| 15:38:12 | gibi | and still not enough | |
| 15:38:32 | gibi | somehow the test VM is pretty IO limited | |
| 15:38:44 | sean-k-mooney | hum well they defalt to 300 second for manila | |
| 15:39:01 | sean-k-mooney | gibi: this only happens on limestone right | |
| 15:39:16 | gibi | the original report said so | |
| 15:39:24 | gibi | but I havent validated that recently | |
| 15:39:38 | sean-k-mooney | im not sure how new the hardware is, we have had io issue in the past where qos was limiting the disk to 100 IOPS on ovh? | |
| 15:45:10 | gibi | sean-k-mooney: now I see more this happening on vexxhost-ca-ymq-1 provider | |
| 15:45:34 | gibi | but maybe it is just because we run test more there | |
| 15:45:38 | gibi | i dont know | |
| 15:47:00 | gibi | hm, here is a neutron migration test with postgres failing for the same reason https://b2b5edd1153190007318-36597f05313fd3fdc073d75819a76269.ssl.cf1.rackcdn.com/766204/7/check/neutron-functional-with-uwsgi/8533ed9/job-output.txt | |
| 15:47:08 | gibi | so it is not at all nova or mysql specific | |
| 15:47:33 | gibi | they have ~ 200sec timeout | |
| 15:47:39 | sean-k-mooney | ya it proably affect all project with a lot of migrations | |
| 15:47:48 | gibi | (nova has 160 for the api db test, and 640 for the nova db test) | |
| 15:47:53 | sean-k-mooney | stephenfin's patch shoudl help with that | |
| 15:48:15 | gibi | the squashing migration one? | |
| 15:48:21 | sean-k-mooney | yep | |
| 15:48:28 | sean-k-mooney | stephenfin: have you pushed that yet ^ | |
| 15:50:17 | gibi | nova meeting starts in 10 minutes... | |
| 15:53:30 | stephenfin | sean-k-mooney: yup, there's a whole series here https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%22+(status:open%20OR%20status:merged) Just waiting for reviews :) | |
| 15:54:36 | gibi | stephenfin: nice | |
| 15:54:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove support for '[DEFAULT] vcpu_pin_set' https://review.opendev.org/c/openstack/nova/+/767576 | |
| 15:54:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Rework some CPU pinning tests to use new config https://review.opendev.org/c/openstack/nova/+/767575 | |
| 15:54:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove '[workarounds] disable_fallback_pcpu_query' https://review.opendev.org/c/openstack/nova/+/767578 | |
| 15:54:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Drop support for pinning of host siblings https://review.opendev.org/c/openstack/nova/+/767577 | |
| 15:54:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Drop VCPU -> PCPU reshape https://review.opendev.org/c/openstack/nova/+/767579 | |
| 15:55:08 | gibi | I can check how much faster the test on my intentionally loaded machine with those compacted migrations | |
| 15:56:00 | stephenfin | sean-k-mooney: Series to drop 'vcpu_pin_set' there ^ | |
| 15:56:34 | stephenfin | I gave up trying to keep the reshaper around and just removed it | |
| 15:57:35 | sean-k-mooney | ok ill try and look at both today i might start with the vcpu one | |
| 16:05:43 | openstackgerrit | Merged openstack/nova master: functional: Add live migration tests for PCI, SR-IOV servers https://review.opendev.org/c/openstack/nova/+/746950 | |
| 16:11:57 | gmann | stephenfin: sean-k-mooney gibi any final direction on this ?, I am holding the code review for this which is very close to merge. https://review.opendev.org/c/openstack/nova-specs/+/765797/1/specs/wallaby/approved/modernize-os-hypervisors-api.rst#205 | |
| 16:12:16 | gibi | gmann: we will touch it today on the meeting | |
| 16:13:11 | gmann | gibi: sorry, forgot about meeting :) thanks for reminder | |
| 16:13:33 | openstackgerrit | Merged openstack/nova master: functional: Expand SR-IOV live migration tests with NUMA https://review.opendev.org/c/openstack/nova/+/749360 | |
| 16:33:35 | martinkennelly | are you in charge of SDN room? | |
| 16:48:24 | gibi | stephenfin: your db migration compating patches make the runtime of those test down by more than 75% | |
| 16:48:42 | stephenfin | hurrah | |
| 16:48:47 | stephenfin | that makes sense tbf | |
| 16:49:12 | stephenfin | there is also an API DB compaction, but I guess those are separate tests | |
| 16:49:32 | gibi | yes, there are unit tests for the nova db and functional test for the api db (dont ask why) | |
| 16:50:00 | gibi | I measured the nova db ones now | |
| 16:50:14 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 16:50:14 | gibi | anyhow your series will eventually solve https://bugs.launchpad.net/nova/+bug/1823251 | |
| 16:50:21 | gibi | I think | |
| 16:50:46 | gibi | but until that I will push timeout increases | |
| 16:57:21 | openstackgerrit | Ghanshyam proposed openstack/nova master: DNM: testing minbw new tests https://review.opendev.org/c/openstack/nova/+/767587 | |
| 17:01:59 | gibi | gmann: good idea ^^ | |
| 17:06:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Increase timeout for db migration tests https://review.opendev.org/c/openstack/nova/+/767590 | |
| 17:07:03 | gibi | sean-k-mooney, stephenfin ^^ | |
| 17:08:15 | stephenfin | LGTM. Do you want to run that a couple of times through CI or merge and let it test itself? | |
| 17:09:21 | gibi | stephenfin: merge it then we can revert if it does not help | |
| 17:09:24 | gibi | :) | |
| 17:09:45 | stephenfin | ack, +2 from moi | |
| 17:09:47 | sean-k-mooney | ah so we increase the scaling factor for the migration and decresss it for other test | |
| 17:10:03 | sean-k-mooney | oh no | |
| 17:10:08 | gibi | sean-k-mooney: nope, it is increase it for all our migration test just different amount | |
| 17:10:12 | sean-k-mooney | we set it to 5 for the main db and 2 for the api | |
| 17:10:17 | gibi | the api db mifration tests were defaulted to 1 | |
| 17:10:31 | sean-k-mooney | right so increased for both | |
| 17:10:33 | sean-k-mooney | ok cool | |
| 17:10:50 | gibi | yepp | |
| 17:11:00 | sean-k-mooney | and you have reference stephens series too | |
| 17:11:04 | sean-k-mooney | ya this looks good to me | |
| 17:11:05 | gibi | if it doesn't really help then I can try to look at serializing them | |
| 17:11:32 | gibi | but it would be better to merge stephen's series instead | |
| 17:12:20 | gibi | anyhow this is the end of my day see you all tomorrow | |
| 17:12:26 | stephenfin | o/ | |
| 17:12:55 | gibi | stephenfin: o/ | |
| 17:57:09 | gmann | gibi: yeah, we can check in advance specially if those new test needs to skip in multi-cell job | |
| 18:21:51 | sean-k-mooney | artom: i see you added your self to https://review.opendev.org/c/openstack/nova/+/767368 | |
| 18:22:19 | sean-k-mooney | that is required before https://review.opendev.org/c/openstack/neutron/+/766277 can merge on the neutron side | |