Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-07-25
12:36:04 opendevreview Balazs Gibizer proposed openstack/nova master: Reject devname based device_spec config https://review.opendev.org/c/openstack/nova/+/846466
12:36:04 opendevreview Balazs Gibizer proposed openstack/nova master: Ignore PCI devs with physical_network tag https://review.opendev.org/c/openstack/nova/+/846219
12:36:06 opendevreview Balazs Gibizer proposed openstack/nova master: Support [pci]device_spec reconfiguration https://review.opendev.org/c/openstack/nova/+/846470
12:36:06 opendevreview Balazs Gibizer proposed openstack/nova master: Stop if tracking is disable after it was enabled before https://review.opendev.org/c/openstack/nova/+/847009
12:36:08 opendevreview Balazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement https://review.opendev.org/c/openstack/nova/+/850468
12:36:08 opendevreview Balazs Gibizer proposed openstack/nova master: Move provider_tree RP creation to PciResourceProvider https://review.opendev.org/c/openstack/nova/+/850546
12:38:34 gibi bauzas: I see 10 hits in the last 7 day: https://paste.opendev.org/show/b8BF5WsTcwJojALnC5J0/ so it is become a bit more frequent than when I reported that bug
12:38:48 bauzas I found 226 hits from the last 7 days
12:42:07 gibi I don't really know how to parse the opensearch query. did you just queried for 'test_live_migration_with_trunk' ? that is all the runs of the test case including when the test passed, isn'tit?
12:42:46 gibi also this 'from:now-30h,to:now' does not seem to be 7 days
12:43:57 gibi I filtered for the nova-next job runs, so my number can be smaller than the global number for sure
12:47:51 gibi bauzas: that is probably closer to the 7 days query of all jobs https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:%22test_live_migration.py%22),sort:!())
12:48:21 bauzas gibi: yeah, I just checked for the testname
12:48:21 gibi I queryd for "test_live_migration.py" do that filters out passing test cases (the test case name is printed ther but not the file)
12:48:34 bauzas as the testname is only provided with a FAILURE
12:48:39 gibi nope
12:48:53 gibi this is a passing test with testname {1} tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_migration_with_trunk [30.171614s] ... ok\
12:49:09 bauzas see the buildrate
12:49:14 bauzas it's 100%
12:49:59 bauzas but agreed, I could query it better
12:51:10 gibi does opensearch filters out SUCCESS runs automatically?
12:51:23 gibi or why the passing runs not appear?
12:51:44 gibi the string "test_live_migration_with_trunk" is in the job-output.txt for passing runs too
12:51:58 gibi so something magic happens in opensearch to filter those
12:53:24 bauzas dunno, just testing this new tool
12:53:38 gibi I don't like magic :D
12:54:10 bauzas well, at least the failure rate seems high and not related to one specific job
12:56:26 gibi yeah I don't think it is related to nova-next at all, I just needed a way to limit my query
12:57:09 gibi my tool don't do a full search on all job results as that would require to download all the job results and logs locally
12:57:13 gibi and that is not feasible
12:58:33 gibi I've just run a widened search on all the nova devstack based jobs, it is 16 hits in 7 days for me and it is hitting nova-next and nova-grenade-multinode in nova
13:05:12 bauzas ovs-hybrid-plug job too
13:17:50 gibi bauzas: good point. I missed that job in my config
13:18:08 gibi this way I get 34 hits in the last 7 days filtered for nova jobs
14:58:30 bauzas I'm getting mad at 1940425
14:58:34 bauzas bug 1940425 I mean
15:07:50 bauzas gibi: wdyt we could do for https://bugs.launchpad.net/nova/+bug/1940425
15:07:57 bauzas we already wait for 60secs
15:08:04 bauzas ?
15:08:28 gibi bauzas: would make sense pinging neutron folks
15:10:00 bauzas failure rate is so high that we can't merge things by now
15:33:48 bauzas mmm, so melwitt had a thought on https://github.com/openstack/tempest/blob/97be23ea6402649652991983f3f2b85873eba4d8/tempest/api/compute/admin/test_live_migration.py#L285 maybr not required for all neutron backends
15:36:32 bauzas slaweq: around ?
15:44:49 melwitt ^ it was sean-k-mooney's thought that I repeated :) sean's the one who knows tons of stuff about nova <--> neutron
15:45:56 bauzas if I'm not getting it wrong, nova isn't only impacted
15:46:06 bauzas this is also hitting cinder
15:59:55 sean-k-mooney melwitt: so the basis of this is that for ovn we only plug one port into ovs. and the trunking is implemented as openflow rules on that port
16:00:08 sean-k-mooney for ml2/ovs and ml2/linux bridge
16:00:19 sean-k-mooney we careate a ovs or linux bridge per trunkport
16:00:53 sean-k-mooney in the ovs case each sub port is create as a patch port pair between the br-int and br-trunk###
16:01:14 sean-k-mooney and the br-int side of the patch port is tagged with the neutron port uuid
16:01:26 sean-k-mooney so ml2/ovs reports the status fo the port as up/active
16:01:35 sean-k-mooney once it finishes wiring them up
16:02:08 sean-k-mooney for ml2/ovn i have no idea if it bothers to report the subports as up since they dont exist on the dataplane level
16:02:11 bauzas sean-k-mooney: except the ovn-hybrid-plug job, this is also failing on nova-next
16:02:20 melwitt I think my brain just exploded from reading that
16:02:43 bauzas ovs-hybrid-plug, my bad
16:03:00 bauzas we also have the grenade-multinode which hits such bug
16:03:05 sean-k-mooney bauzas: ack so its failing regardesll of how its implemneted
16:03:08 gibi the interesting part that it is not 100% failure in nova-next, so in some case neutron does bring up the subport
16:03:17 bauzas yup
16:03:21 sean-k-mooney it could be a timing thing
16:03:23 bauzas this looks a transient issue
16:03:32 bauzas we already wait for 60s
16:03:33 sean-k-mooney neutron is not ment to send network vif pulgged for the parent
16:03:33 gibi but 60sec is a lot
16:03:39 sean-k-mooney untill all the subports are setup
16:03:46 sean-k-mooney maybe it does nto take that into account
16:04:00 sean-k-mooney and sends it only once the parent is set up
16:04:06 sean-k-mooney meaning we might be racing
16:04:16 gibi yeah that make sense
16:04:25 sean-k-mooney nova only has one port attached to the vm so we only care about the parent
16:04:28 gibi that will depend on how was tempest querying the state
16:05:01 sean-k-mooney i dont think the parent shoudl really be active if the subports are not active
16:05:03 melwitt this is the tempest test https://github.com/openstack/tempest/blob/97be23ea6402649652991983f3f2b85873eba4d8/tempest/api/compute/admin/test_live_migration.py#L285
16:05:16 sean-k-mooney but honestly that is proably an impmentation detail that noone should depend on
16:05:26 sean-k-mooney i dont think this was defiended in teh specs
16:05:59 sean-k-mooney so the test might just be asserting stuff that is not required/guarenteed by the api
16:07:28 sean-k-mooney im pretty sure this was the relevent spec https://specs.openstack.org/openstack/neutron-specs/specs/newton/vlan-aware-vms.html
16:10:00 sean-k-mooney gibi: melwitt so ya reading that quickly the status of the subport is not defeined in relation to port binding
16:10:18 dansmith sean-k-mooney: on the event, it should send the event once the thing the port represents can pass traffic, right? so subport or not, we shouldn't get the alert until the traffic will flow
16:10:32 dansmith else we're wiring up to something we can't expect to get dhcp or other critical traffic through
16:10:36 sean-k-mooney correct
16:10:47 dansmith calling the trunk up because one side is active isn't good enough
16:11:00 sean-k-mooney we should not get the network-vif-plugged for the trunk parent untill everything is configured to allow all taffic to flow
16:11:26 dansmith right, so depending on the backend implementation that may come depending on what wiring needs to happen
16:11:30 sean-k-mooney but they may not have implemented that depened status check
16:11:47 opendevreview Balazs Gibizer proposed openstack/nova master: Add more test coverage for devname base dev spec https://review.opendev.org/c/openstack/nova/+/844625
16:11:48 opendevreview Balazs Gibizer proposed openstack/nova master: Unparent PciDeviceSpec from PciAddressSpec https://review.opendev.org/c/openstack/nova/+/844491
16:11:48 opendevreview Balazs Gibizer proposed openstack/nova master: Extra tests for remote managed dev spec https://review.opendev.org/c/openstack/nova/+/844626
16:11:49 opendevreview Balazs Gibizer proposed openstack/nova master: Remove dead code from PhysicalPciAddress https://review.opendev.org/c/openstack/nova/+/844628
16:11:49 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PciAddressSpec descendants to call super.__init__ https://review.opendev.org/c/openstack/nova/+/844565
16:11:50 opendevreview Balazs Gibizer proposed openstack/nova master: Remove unused PF checking from get_function_by_ifname https://review.opendev.org/c/openstack/nova/+/845775
16:11:50 dansmith right but that would be a neutron issue
16:11:50 opendevreview Balazs Gibizer proposed openstack/nova master: Clean up mapping input to address spec types https://review.opendev.org/c/openstack/nova/+/845765
16:11:51 opendevreview Balazs Gibizer proposed openstack/nova master: Move __str__ to the PciAddressSpec base class https://review.opendev.org/c/openstack/nova/+/845781
16:11:51 opendevreview Balazs Gibizer proposed openstack/nova master: Fix type annotation of pci.Whitelist class https://review.opendev.org/c/openstack/nova/+/845780
16:11:58 sean-k-mooney dansmith: yes it would
16:12:23 dansmith ack, just confirming ;)
16:12:31 sean-k-mooney so your suggesting that marking it confirmed for nova on the bug is wrong and we should likely change it

Earlier   Later