| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-17 | |||
| 15:37:38 | sean-k-mooney | you mentioned something about scale factor in the bug | |
| 15:37:53 | gibi | yeah, we have a 160sec per test case timeout | |
| 15:37:54 | sean-k-mooney | ah self.useFixture(timeout.Timeout( | |
| 15:37:55 | sean-k-mooney | scaling_factor=self.TIMEOUT_SCALING_FACTOR)) | |
| 15:38:08 | gibi | but the mysql tests uses 4 * 160 sec | |
| 15:38:12 | gibi | and still not enough | |
| 15:38:32 | gibi | somehow the test VM is pretty IO limited | |
| 15:38:44 | sean-k-mooney | hum well they defalt to 300 second for manila | |
| 15:39:01 | sean-k-mooney | gibi: this only happens on limestone right | |
| 15:39:16 | gibi | the original report said so | |
| 15:39:24 | gibi | but I havent validated that recently | |
| 15:39:38 | sean-k-mooney | im not sure how new the hardware is, we have had io issue in the past where qos was limiting the disk to 100 IOPS on ovh? | |
| 15:45:10 | gibi | sean-k-mooney: now I see more this happening on vexxhost-ca-ymq-1 provider | |
| 15:45:34 | gibi | but maybe it is just because we run test more there | |
| 15:45:38 | gibi | i dont know | |
| 15:47:00 | gibi | hm, here is a neutron migration test with postgres failing for the same reason https://b2b5edd1153190007318-36597f05313fd3fdc073d75819a76269.ssl.cf1.rackcdn.com/766204/7/check/neutron-functional-with-uwsgi/8533ed9/job-output.txt | |
| 15:47:08 | gibi | so it is not at all nova or mysql specific | |
| 15:47:33 | gibi | they have ~ 200sec timeout | |
| 15:47:39 | sean-k-mooney | ya it proably affect all project with a lot of migrations | |
| 15:47:48 | gibi | (nova has 160 for the api db test, and 640 for the nova db test) | |
| 15:47:53 | sean-k-mooney | stephenfin's patch shoudl help with that | |
| 15:48:15 | gibi | the squashing migration one? | |
| 15:48:21 | sean-k-mooney | yep | |
| 15:48:28 | sean-k-mooney | stephenfin: have you pushed that yet ^ | |
| 15:50:17 | gibi | nova meeting starts in 10 minutes... | |
| 15:53:30 | stephenfin | sean-k-mooney: yup, there's a whole series here https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%22+(status:open%20OR%20status:merged) Just waiting for reviews :) | |
| 15:54:36 | gibi | stephenfin: nice | |
| 15:54:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove support for '[DEFAULT] vcpu_pin_set' https://review.opendev.org/c/openstack/nova/+/767576 | |
| 15:54:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Rework some CPU pinning tests to use new config https://review.opendev.org/c/openstack/nova/+/767575 | |
| 15:54:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove '[workarounds] disable_fallback_pcpu_query' https://review.opendev.org/c/openstack/nova/+/767578 | |
| 15:54:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Drop support for pinning of host siblings https://review.opendev.org/c/openstack/nova/+/767577 | |
| 15:54:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Drop VCPU -> PCPU reshape https://review.opendev.org/c/openstack/nova/+/767579 | |
| 15:55:08 | gibi | I can check how much faster the test on my intentionally loaded machine with those compacted migrations | |
| 15:56:00 | stephenfin | sean-k-mooney: Series to drop 'vcpu_pin_set' there ^ | |
| 15:56:34 | stephenfin | I gave up trying to keep the reshaper around and just removed it | |
| 15:57:35 | sean-k-mooney | ok ill try and look at both today i might start with the vcpu one | |
| 16:05:43 | openstackgerrit | Merged openstack/nova master: functional: Add live migration tests for PCI, SR-IOV servers https://review.opendev.org/c/openstack/nova/+/746950 | |
| 16:11:57 | gmann | stephenfin: sean-k-mooney gibi any final direction on this ?, I am holding the code review for this which is very close to merge. https://review.opendev.org/c/openstack/nova-specs/+/765797/1/specs/wallaby/approved/modernize-os-hypervisors-api.rst#205 | |
| 16:12:16 | gibi | gmann: we will touch it today on the meeting | |
| 16:13:11 | gmann | gibi: sorry, forgot about meeting :) thanks for reminder | |
| 16:13:33 | openstackgerrit | Merged openstack/nova master: functional: Expand SR-IOV live migration tests with NUMA https://review.opendev.org/c/openstack/nova/+/749360 | |
| 16:33:35 | martinkennelly | are you in charge of SDN room? | |
| 16:48:24 | gibi | stephenfin: your db migration compating patches make the runtime of those test down by more than 75% | |
| 16:48:42 | stephenfin | hurrah | |
| 16:48:47 | stephenfin | that makes sense tbf | |
| 16:49:12 | stephenfin | there is also an API DB compaction, but I guess those are separate tests | |
| 16:49:32 | gibi | yes, there are unit tests for the nova db and functional test for the api db (dont ask why) | |
| 16:50:00 | gibi | I measured the nova db ones now | |
| 16:50:14 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 16:50:14 | gibi | anyhow your series will eventually solve https://bugs.launchpad.net/nova/+bug/1823251 | |
| 16:50:21 | gibi | I think | |
| 16:50:46 | gibi | but until that I will push timeout increases | |
| 16:57:21 | openstackgerrit | Ghanshyam proposed openstack/nova master: DNM: testing minbw new tests https://review.opendev.org/c/openstack/nova/+/767587 | |
| 17:01:59 | gibi | gmann: good idea ^^ | |
| 17:06:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Increase timeout for db migration tests https://review.opendev.org/c/openstack/nova/+/767590 | |
| 17:07:03 | gibi | sean-k-mooney, stephenfin ^^ | |
| 17:08:15 | stephenfin | LGTM. Do you want to run that a couple of times through CI or merge and let it test itself? | |
| 17:09:21 | gibi | stephenfin: merge it then we can revert if it does not help | |
| 17:09:24 | gibi | :) | |
| 17:09:45 | stephenfin | ack, +2 from moi | |
| 17:09:47 | sean-k-mooney | ah so we increase the scaling factor for the migration and decresss it for other test | |
| 17:10:03 | sean-k-mooney | oh no | |
| 17:10:08 | gibi | sean-k-mooney: nope, it is increase it for all our migration test just different amount | |
| 17:10:12 | sean-k-mooney | we set it to 5 for the main db and 2 for the api | |
| 17:10:17 | gibi | the api db mifration tests were defaulted to 1 | |
| 17:10:31 | sean-k-mooney | right so increased for both | |
| 17:10:33 | sean-k-mooney | ok cool | |
| 17:10:50 | gibi | yepp | |
| 17:11:00 | sean-k-mooney | and you have reference stephens series too | |
| 17:11:04 | sean-k-mooney | ya this looks good to me | |
| 17:11:05 | gibi | if it doesn't really help then I can try to look at serializing them | |
| 17:11:32 | gibi | but it would be better to merge stephen's series instead | |
| 17:12:20 | gibi | anyhow this is the end of my day see you all tomorrow | |
| 17:12:26 | stephenfin | o/ | |
| 17:12:55 | gibi | stephenfin: o/ | |
| 17:57:09 | gmann | gibi: yeah, we can check in advance specially if those new test needs to skip in multi-cell job | |
| 18:21:51 | sean-k-mooney | artom: i see you added your self to https://review.opendev.org/c/openstack/nova/+/767368 | |
| 18:22:19 | sean-k-mooney | that is required before https://review.opendev.org/c/openstack/neutron/+/766277 can merge on the neutron side | |
| 18:22:31 | sean-k-mooney | and we need to backport it to at least train | |
| 18:23:00 | sean-k-mooney | its related to https://review.opendev.org/c/openstack/nova/+/602432 also | |
| 18:23:32 | sean-k-mooney | we are currenly up to 6 pathces we need to backport to fix live migration | |
| 18:23:39 | sean-k-mooney | 3 on nova and 3 on neutron | |
| 18:24:06 | sean-k-mooney | 2 are merged on the neutron side already and 1 on the nova side | |
| 18:24:28 | sean-k-mooney | if we need to go before train we will need to backport thing in os-vif also | |
| 18:25:25 | sean-k-mooney | anyway im going to be calling it a day soon then going on PTO till january so ill respond it comments when im back if there are any | |
| 18:27:47 | sean-k-mooney | gibi: just saw this in the conflicts list https://review.opendev.org/c/openstack/nova/+/709658 we proably should arcive that just to not distract peopel with it | |
| 18:29:41 | sean-k-mooney | oh enjoy the break all o/ i may or may not be around over the next 2 weeks but im on pto until the 4th of january | |
| 18:55:16 | lyarwood | \o enjoy either way, catch you in the new year | |
| 19:10:14 | artom | sean-k-mooney, yeah, seemed like a thing I need to look at | |
| 19:10:43 | artom | (Sorry for the lag, was doing some errands and getting a haircut, before Shutdown 2 Electric Boogalloo after Christmas) | |
| 19:11:04 | sean-k-mooney | no worries | |
| 19:11:38 | sean-k-mooney | tl;dr it only every worked because of buggy behavior in neutron that when fix means we need to actully wait properly in nova | |
| 19:12:02 | sean-k-mooney | basically we were migration before because fo a network-vif-plugged event form the dhcp agent | |
| 19:12:13 | sean-k-mooney | not form the l2 agent that we were waiting for | |
| 19:12:30 | artom | sean-k-mooney, I suspect this'll be another "Artom and Sean spent 2 hours on GMeet" kind of situation :) | |
| 19:12:54 | sean-k-mooney | about an hour and with radolfo but ya | |
| 19:13:23 | sean-k-mooney | i had a suption this was true for a while but we found the atcual code that was at fault friday last week | |
| 19:14:16 | sean-k-mooney | we have had a much of unxepcect event in the logs for a few releases and this is why many of them happened | |
| 19:14:42 | sean-k-mooney | its all related to the multipe port binding work in rocky | |
| 22:12:21 | openstackgerrit | Ghanshyam proposed openstack/placement master: Remove deprecated [placement]/policy_file config option https://review.opendev.org/c/openstack/placement/+/764409 | |