| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-25 | |||
| 11:50:50 | sean-k-mooney | but first party ci will run all jobs together | |
| 11:54:07 | auniyal_ | earlier I saw somewhere, that we should not run all jobs if only 1 or 2 job fails and can only run by single jobs using recheck |
|
| 12:08:40 | sean-k-mooney | the confirution is per pipeline and we expeictly do not allow that in openstack | |
| 12:09:04 | sean-k-mooney | zuul may support that but we do not allow that in openstack under the green check policy | |
| 12:09:23 | sean-k-mooney | all jobs on the check run must use the same revison fo the code | |
| 12:09:33 | sean-k-mooney | if you recheck indivicual jobs that woudl not be the case | |
| 12:31:01 | bauzas | folks, we have a problem with tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_migration_with_trunk | |
| 12:35:12 | bauzas | https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30h,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:test_live_migration_with_trunk),sort:!()) | |
| 12:35:54 | bauzas | looks like it's https://bugs.launchpad.net/nova/+bug/1940425 | |
| 12:35:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add compute restart capability for libvirt func tests https://review.opendev.org/c/openstack/nova/+/850510 | |
| 12:35:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Poison /sys access via various calls in test https://review.opendev.org/c/openstack/nova/+/844627 | |
| 12:36:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rename exception.PciConfigInvalidWhitelist to PciConfigInvalidSpec https://review.opendev.org/c/openstack/nova/+/843861 | |
| 12:36:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rename [pci]passthrough_whitelist to device_spec https://review.opendev.org/c/openstack/nova/+/843834 | |
| 12:36:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Basics for PCI Placement reporting https://review.opendev.org/c/openstack/nova/+/846187 | |
| 12:36:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rename whitelist in tests https://review.opendev.org/c/openstack/nova/+/843862 | |
| 12:36:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reject PCI dependent device config https://review.opendev.org/c/openstack/nova/+/846435 | |
| 12:36:02 | opendevreview | Balazs Gibizer proposed openstack/nova master: Extend device_spec with resource_class and traits https://review.opendev.org/c/openstack/nova/+/846218 | |
| 12:36:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reject mixed VF rc and trait config https://review.opendev.org/c/openstack/nova/+/846436 | |
| 12:36:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reject devname based device_spec config https://review.opendev.org/c/openstack/nova/+/846466 | |
| 12:36:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Ignore PCI devs with physical_network tag https://review.opendev.org/c/openstack/nova/+/846219 | |
| 12:36:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support [pci]device_spec reconfiguration https://review.opendev.org/c/openstack/nova/+/846470 | |
| 12:36:06 | opendevreview | Balazs Gibizer proposed openstack/nova master: Stop if tracking is disable after it was enabled before https://review.opendev.org/c/openstack/nova/+/847009 | |
| 12:36:08 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement https://review.opendev.org/c/openstack/nova/+/850468 | |
| 12:36:08 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move provider_tree RP creation to PciResourceProvider https://review.opendev.org/c/openstack/nova/+/850546 | |
| 12:38:34 | gibi | bauzas: I see 10 hits in the last 7 day: https://paste.opendev.org/show/b8BF5WsTcwJojALnC5J0/ so it is become a bit more frequent than when I reported that bug | |
| 12:38:48 | bauzas | I found 226 hits from the last 7 days | |
| 12:42:07 | gibi | I don't really know how to parse the opensearch query. did you just queried for 'test_live_migration_with_trunk' ? that is all the runs of the test case including when the test passed, isn'tit? | |
| 12:42:46 | gibi | also this 'from:now-30h,to:now' does not seem to be 7 days | |
| 12:43:57 | gibi | I filtered for the nova-next job runs, so my number can be smaller than the global number for sure | |
| 12:47:51 | gibi | bauzas: that is probably closer to the 7 days query of all jobs https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:%22test_live_migration.py%22),sort:!()) | |
| 12:48:21 | bauzas | gibi: yeah, I just checked for the testname | |
| 12:48:21 | gibi | I queryd for "test_live_migration.py" do that filters out passing test cases (the test case name is printed ther but not the file) | |
| 12:48:34 | bauzas | as the testname is only provided with a FAILURE | |
| 12:48:39 | gibi | nope | |
| 12:48:53 | gibi | this is a passing test with testname {1} tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_migration_with_trunk [30.171614s] ... ok\ | |
| 12:49:09 | bauzas | see the buildrate | |
| 12:49:14 | bauzas | it's 100% | |
| 12:49:59 | bauzas | but agreed, I could query it better | |
| 12:51:10 | gibi | does opensearch filters out SUCCESS runs automatically? | |
| 12:51:23 | gibi | or why the passing runs not appear? | |
| 12:51:44 | gibi | the string "test_live_migration_with_trunk" is in the job-output.txt for passing runs too | |
| 12:51:58 | gibi | so something magic happens in opensearch to filter those | |
| 12:53:24 | bauzas | dunno, just testing this new tool | |
| 12:53:38 | gibi | I don't like magic :D | |
| 12:54:10 | bauzas | well, at least the failure rate seems high and not related to one specific job | |
| 12:56:26 | gibi | yeah I don't think it is related to nova-next at all, I just needed a way to limit my query | |
| 12:57:09 | gibi | my tool don't do a full search on all job results as that would require to download all the job results and logs locally | |
| 12:57:13 | gibi | and that is not feasible | |
| 12:58:33 | gibi | I've just run a widened search on all the nova devstack based jobs, it is 16 hits in 7 days for me and it is hitting nova-next and nova-grenade-multinode in nova | |
| 13:05:12 | bauzas | ovs-hybrid-plug job too | |
| 13:17:50 | gibi | bauzas: good point. I missed that job in my config | |
| 13:18:08 | gibi | this way I get 34 hits in the last 7 days filtered for nova jobs | |
| 14:58:30 | bauzas | I'm getting mad at 1940425 | |
| 14:58:34 | bauzas | bug 1940425 I mean | |
| 15:07:50 | bauzas | gibi: wdyt we could do for https://bugs.launchpad.net/nova/+bug/1940425 | |
| 15:07:57 | bauzas | we already wait for 60secs | |
| 15:08:04 | bauzas | ? | |
| 15:08:28 | gibi | bauzas: would make sense pinging neutron folks | |
| 15:10:00 | bauzas | failure rate is so high that we can't merge things by now | |
| 15:33:48 | bauzas | mmm, so melwitt had a thought on https://github.com/openstack/tempest/blob/97be23ea6402649652991983f3f2b85873eba4d8/tempest/api/compute/admin/test_live_migration.py#L285 maybr not required for all neutron backends | |
| 15:36:32 | bauzas | slaweq: around ? | |
| 15:44:49 | melwitt | ^ it was sean-k-mooney's thought that I repeated :) sean's the one who knows tons of stuff about nova <--> neutron | |
| 15:45:56 | bauzas | if I'm not getting it wrong, nova isn't only impacted | |
| 15:46:06 | bauzas | this is also hitting cinder | |
| 15:59:55 | sean-k-mooney | melwitt: so the basis of this is that for ovn we only plug one port into ovs. and the trunking is implemented as openflow rules on that port | |
| 16:00:08 | sean-k-mooney | for ml2/ovs and ml2/linux bridge | |
| 16:00:19 | sean-k-mooney | we careate a ovs or linux bridge per trunkport | |
| 16:00:53 | sean-k-mooney | in the ovs case each sub port is create as a patch port pair between the br-int and br-trunk### | |
| 16:01:14 | sean-k-mooney | and the br-int side of the patch port is tagged with the neutron port uuid | |
| 16:01:26 | sean-k-mooney | so ml2/ovs reports the status fo the port as up/active | |
| 16:01:35 | sean-k-mooney | once it finishes wiring them up | |
| 16:02:08 | sean-k-mooney | for ml2/ovn i have no idea if it bothers to report the subports as up since they dont exist on the dataplane level | |
| 16:02:11 | bauzas | sean-k-mooney: except the ovn-hybrid-plug job, this is also failing on nova-next | |
| 16:02:20 | melwitt | I think my brain just exploded from reading that | |
| 16:02:43 | bauzas | ovs-hybrid-plug, my bad | |
| 16:03:00 | bauzas | we also have the grenade-multinode which hits such bug | |
| 16:03:05 | sean-k-mooney | bauzas: ack so its failing regardesll of how its implemneted | |
| 16:03:08 | gibi | the interesting part that it is not 100% failure in nova-next, so in some case neutron does bring up the subport | |
| 16:03:17 | bauzas | yup | |
| 16:03:21 | sean-k-mooney | it could be a timing thing | |
| 16:03:23 | bauzas | this looks a transient issue | |
| 16:03:32 | bauzas | we already wait for 60s | |
| 16:03:33 | sean-k-mooney | neutron is not ment to send network vif pulgged for the parent | |
| 16:03:33 | gibi | but 60sec is a lot | |
| 16:03:39 | sean-k-mooney | untill all the subports are setup | |
| 16:03:46 | sean-k-mooney | maybe it does nto take that into account | |
| 16:04:00 | sean-k-mooney | and sends it only once the parent is set up | |
| 16:04:06 | sean-k-mooney | meaning we might be racing | |
| 16:04:16 | gibi | yeah that make sense | |
| 16:04:25 | sean-k-mooney | nova only has one port attached to the vm so we only care about the parent | |
| 16:04:28 | gibi | that will depend on how was tempest querying the state | |
| 16:05:01 | sean-k-mooney | i dont think the parent shoudl really be active if the subports are not active | |
| 16:05:03 | melwitt | this is the tempest test https://github.com/openstack/tempest/blob/97be23ea6402649652991983f3f2b85873eba4d8/tempest/api/compute/admin/test_live_migration.py#L285 | |
| 16:05:16 | sean-k-mooney | but honestly that is proably an impmentation detail that noone should depend on | |
| 16:05:26 | sean-k-mooney | i dont think this was defiended in teh specs | |
| 16:05:59 | sean-k-mooney | so the test might just be asserting stuff that is not required/guarenteed by the api | |
| 16:07:28 | sean-k-mooney | im pretty sure this was the relevent spec https://specs.openstack.org/openstack/neutron-specs/specs/newton/vlan-aware-vms.html | |
| 16:10:00 | sean-k-mooney | gibi: melwitt so ya reading that quickly the status of the subport is not defeined in relation to port binding | |
| 16:10:18 | dansmith | sean-k-mooney: on the event, it should send the event once the thing the port represents can pass traffic, right? so subport or not, we shouldn't get the alert until the traffic will flow | |
| 16:10:32 | dansmith | else we're wiring up to something we can't expect to get dhcp or other critical traffic through | |