Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-21
15:57:44 opendevreview ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106
15:57:44 opendevreview ribaudr proposed openstack/nova master: FUP: Improve libvirt fixture for hostdevs https://review.opendev.org/c/openstack/nova/+/945232
18:14:46 dansmith vmware ci has been broken for quite some time it feels
18:15:17 dansmith the run I just checked seems to be failing with volume backed rebuild, which has been a thing for what two cycles?
20:10:17 dansmith melwitt: this works for me, if I push it up would you +W if I +2 even though it's my own work? https://termbin.com/cpvj
20:13:53 sean-k-mooney dansmith: i dont think they every got that to work
20:14:07 dansmith sean-k-mooney: what?
20:14:10 sean-k-mooney dansmith: i know we have not merged all the patches they submited to fix thing
20:14:10 dansmith who?
20:14:17 sean-k-mooney dansmith: the vmware ci
20:14:20 dansmith ah
20:15:10 sean-k-mooney i belive the rebuild thign is one of the pendign patches but ill admin i have not looked at the vmware patches in like 4 months
20:15:28 dansmith okay
20:19:11 melwitt dansmith: I wasn't saying there's anything wrong or need to be changed fwiw, just pointing it out and have bogdan ack it if he agrees
20:19:55 melwitt (if I understood his comment right anyway)
20:21:52 dansmith melwitt: okay was just trying to keep this moving, but fair enough
20:22:10 dansmith I'm glad you were able to distill something out of that because I was highly confused
20:23:55 melwitt ack
20:25:57 sean-k-mooney dansmith: https://review.opendev.org/c/openstack/nova/+/941095 i think is the fix for https://bugs.launchpad.net/cinder/+bug/2097771
20:26:17 sean-k-mooney or https://bugs.launchpad.net/nova/+bug/2058225
20:26:21 sean-k-mooney depending on you perspective
20:26:57 sean-k-mooney but its failing tempest with vmware so http://openstack-ci-logs.global.cloud.sap/openstack-nova-941095-4ptpq/tempest.html
20:35:06 dansmith hrm
20:35:37 dansmith so that's been broken and WIP for like a year
20:36:47 sean-k-mooney i think the current issue is just the key/passwrod
20:37:04 sean-k-mooney i know that they had some neutron networking issues for a while
20:50:29 sean-k-mooney dansmith: im kind of torn, on one hand we didnt spend a lot of time reviewign the vmware fixes
20:50:53 sean-k-mooney dansmith: on the other hand not much progress has been made since we paused the removal but it also has not got worse
20:51:03 sean-k-mooney so its still kind of limping along
20:52:11 opendevreview sean mooney proposed openstack/nova master: wrap wsgi_app.init_application with latch_error_on_raise https://review.opendev.org/c/openstack/nova/+/945028
20:55:20 sean-k-mooney dansmith: i slightly tweaked the testing and fixed a comment if you can readd +2
20:55:42 sean-k-mooney im goign to kick of one more run of the operator job and call it a day
20:55:52 sean-k-mooney we can tak a look at the resulst on monday
21:34:12 dansmith it's also wip, so not really a candidate anyway
#openstack-nova - 2025-03-24
15:35:46 dansmith gibi: sean-k-mooney: so starting to look at placement where the capacity checks happen, for addressing the overcommit bugs...
15:36:12 sean-k-mooney oh fun
15:36:15 dansmith I'm only seeing a single test that ever asserts the capacity-exceeded scenario in all of placement's unit and functional tests
15:36:38 dansmith that is, like, unexpected since that's kinda the core function of placement
15:36:58 sean-k-mooney when you say functional did you include the gabbit tests
15:36:59 dansmith I know there are lots of gabbit tests I'm probably not grepping to find, but.. am I missing something?
15:37:21 sean-k-mooney well the gabbit test are the primarly api test/funcitonal tests in placment
15:37:44 sean-k-mooney i know they are closer to neutron fullstack tests then our functional tests
15:37:59 sean-k-mooney but i think thats where a lot of the end to end functial behaivor is verifed
15:38:24 sean-k-mooney https://github.com/openstack/placement/tree/master/placement/tests/functional/gabbits
15:39:01 sean-k-mooney there are a number of allcoation an dallcoation bug tests there for sepcifc beahivor
15:39:58 dansmith yeah,
15:40:13 dansmith but only one that explicitly tests overcapacity for allocations, AFAICT,
15:40:16 sean-k-mooney dansmith: i belive some of this is also enfoced at the db level althouhg i dont think they are usign db triggers or anything to actully encode it at the db level
15:40:21 dansmith and it's just the simplest case of asking for more than exists
15:40:32 dansmith sean-k-mooney: it is, and there's a single db test for one simple sccenario there
15:40:40 sean-k-mooney ack
15:40:50 dansmith I mean the db object layer in placement
15:41:00 sean-k-mooney i guess the assumtin is if you neever allow a simple allcotion to oversubscibe that covers the more advanced cases
15:41:03 dansmith there's a big complex function that checks if we're over capacity after an allocation
15:41:12 sean-k-mooney right
15:41:31 sean-k-mooney so https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-bug-1779717.yaml#L73 is testing creating an allcotion for multiple consumers
15:41:38 dansmith it's not that simple though, that big function has a lot of edges that can't possibly be covered by a single test (single python test and a single gabbit test)
15:41:47 dansmith yep
15:42:07 sean-k-mooney although that is not really about capastiy
15:42:16 sean-k-mooney lookign at the bug itself https://bugs.launchpad.net/nova/+bug/1779717
15:43:15 sean-k-mooney i guess to un pick this a littel you might want to create a repoducer test for the current evac/cold migration case
15:43:21 sean-k-mooney where we are doing the swap
15:43:38 sean-k-mooney i suspect you can do that without gabit using the placement fixture
15:43:57 dansmith well, I want to test the underlying mechanics in placement, not nova
15:44:03 sean-k-mooney we certenly could do that in nova so i dont see why you could not in placment itself
15:44:09 dansmith and I expected to see a lot more coverage in placement given how complex the capacity check is
15:44:30 sean-k-mooney right but the the actial api behavior does not actully depend on nova
15:44:48 sean-k-mooney you jsut need to create the allcoation mess with reserved then try and swap it
15:45:01 sean-k-mooney basiclly emulate the OTU flow
15:45:32 sean-k-mooney i know you know that by the way im just trying to state the repoducer clearly
15:46:44 sean-k-mooney gabit is expressive enough to encode that but its not very deep coverage hench why you probly will want to try using the placement fixutre and writign a new functional test
15:46:59 dansmith yeah I'm very aware of what I need to do, I'm just surprised that there is so little coverage of this _area_ in placement
15:47:33 sean-k-mooney ill fully adming i have never been that deep on the implemation side of placement
15:47:39 sean-k-mooney only the usage of it
15:47:48 sean-k-mooney so i havent really looked tat the testing in any detail
15:47:52 dansmith okay I figured you and gibi are the most familiar, but maybe not
15:47:55 dansmith ack
15:48:25 sean-k-mooney gibi and melwitt i think. api wise yes but ptyhon wise less so on my part
15:48:50 sean-k-mooney im willing to review placement code but i generally need to learn the context every time i do
15:49:28 gibi dansmith: I would expect more than one over capacity test case. let me take a look
15:49:56 sean-k-mooney i think tis is the only functional testing of allctions https://github.com/openstack/placement/blob/master/placement/tests/functional/test_allocation.py
15:50:02 sean-k-mooney i.e. in python
15:50:10 dansmith I definitely want to do this in python functional tests and not gabbits, FWIW
15:51:20 sean-k-mooney dansmith: this is the one you wer referint too for the db level right https://github.com/openstack/placement/blob/master/placement/tests/functional/db/test_allocation.py#L519
15:51:51 sean-k-mooney a number of those functional tests are really like 5 tests in one
15:52:00 dansmith yup
15:52:24 dansmith yup, referring, but not sure yup in terms of coverage
15:52:30 dansmith the capacity check is very complex :)
15:53:20 sean-k-mooney https://github.com/search?q=repo%3Aopenstack/placement%20InvalidAllocationCapacityExceeded&type=code not that i realy turst github search but only 3 hits total
15:53:49 sean-k-mooney codesearch agrees
15:54:18 sean-k-mooney that excetion is not used elsewhere in the codebase
15:55:54 sean-k-mooney i guess there is also InvalidInventoryCapacity
15:56:18 sean-k-mooney but i think thats raised when you modify the inventory
15:56:38 dansmith right
15:56:52 gibi this file has some POST allocation tests https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-post.yaml
15:57:19 gibi but I understand you want moe
15:57:20 gibi more
15:57:43 sean-k-mooney right and that does https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/allocations-post.yaml#L212-L236
15:58:27 dansmith gibi: yeah it's not just allocation, it's allocation with the capacity check
15:58:35 dansmith *failing

Earlier   Later