Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-08-31
16:04:59 gibi yepp
16:05:12 bauzas ie. something that worked before Xena but now has issues with this release
16:05:37 bauzas we only have 2 weeks before RC1 in order to merge bugfixes that *aren't* regressions
16:05:49 bauzas or those would be fixed in the next Yoga release
16:05:59 gibi and backported :)
16:06:09 sean-k-mooney i dont think we currenlty have any critical bugs that would need the rc flag
16:06:13 bauzas so, pings are appreciated besides classic feature review requests
16:06:30 bauzas we have an interesting race condition issue with placement
16:06:41 gibi yepp I saved that for the gate issue topic
16:06:51 gibi lets go there
16:06:52 bauzas this isn't only impacting the gate, right?
16:07:20 gibi it is a new race so so far we only have infor from the gate about it
16:07:23 bauzas but yeah, let's discuss this bug on the gate section
16:07:28 gibi #topic Gate status
16:07:32 gibi Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure
16:07:41 gibi We had the allocation deletion conflict bug #link https://bugs.launchpad.net/nova/+bug/1836754 that hopefully resolved when we landed #link https://review.opendev.org/c/openstack/nova/+/688802 yesterday
16:08:05 gibi We have a project creation race in allocation update bug as well #link https://storyboard.openstack.org/#!/story/2009159 that is affecting the those gate jobs that are booting more than on VM at the same time. Octavia is heavily affected. I've pushed a reproducer #link https://review.opendev.org/c/openstack/placement/+/806730 but I don't know how to fix the actual bug.
16:08:40 gibi this is the one bauzas mentioned above ^^
16:09:30 bauzas if the race isn't occurring a lot of times, we should maybe just call the transaction again
16:09:37 gibi in short if two VM for the same project is scheduled at the same time then the PUT /allocations call could cause a race in the project creation in placement
16:10:16 gibi bauzas: that was my first idea, but looking the code I'm not sure about the amount of surgery needed for this
16:10:33 dansmith it must be the *first* two vms right?
16:10:34 gibi if somebody has deeper sqlAlchemy knowledge than me then help is appreciated
16:10:45 bauzas dansmith: it *should* be yeah
16:10:47 gibi dansmith: first two VMs for a project yes
16:10:55 sean-k-mooney gibi: sorry im a bit confusted by the term project createion in a placment context
16:10:59 dansmith that definitely seems uncool
16:11:15 bauzas I guess we could hit the problem often with multicreate
16:11:30 gibi sean-k-mooney: placement stores the project_id and user_id of each consumer so that usages can be aggregated per project
16:11:30 bauzas ie. server create --max 2
16:11:38 sean-k-mooney why does placement need to be project wawre beyond storigng the project/user id in the allocation?
16:11:54 bauzas sean-k-mooney: because we use it from an API query
16:12:01 dansmith sean-k-mooney: it's relational so it has a table for it I think
16:12:07 sean-k-mooney yes im aware it there for usage
16:12:11 bauzas you can filter per project
16:12:17 gibi https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages
16:12:19 sean-k-mooney but im not sure what resouce in the db we would need to create that would race
16:12:21 bauzas this was my question earlier this EU morning time :)
16:12:34 gibi sean-k-mooney: it is the row in the projects table
16:12:35 dansmith sean-k-mooney: the first time that entry goes into the table for the project?
16:12:43 gibi yes ^^
16:12:48 bauzas correct
16:12:54 sean-k-mooney right im wondering why we need a proejct tbale at all
16:13:09 dansmith ...because it's relational :)
16:13:11 gibi sean-k-mooney: for the /usages?project_id= query
16:13:12 sean-k-mooney i assuem its just for forine key constriants
16:13:28 sean-k-mooney dansmith: right but we dont actuly do that in other dbs in openstack
16:13:41 dansmith we could create the project first in a separate transaction, which is easy to retry, but that's definitely surgery
16:13:47 dansmith sean-k-mooney: not because it's a good idea :)
16:14:10 gibi dansmith: actually I think that is what happened before consumer_types
16:14:47 gibi dansmith: project_id user_id and consumer creation was each in a separate transaction, then a single transaction created / updated the allocation
16:15:04 dansmith gibi: ah, so this is more fallout from that work?
16:15:09 gibi it seems so
16:15:16 dansmith okay
16:15:22 gibi dansmith: this changed, now the whole PUT /allocations is a single DB transaction
16:15:40 dansmith for something like the project record, it makes sense to do that in a separate transaction I think
16:15:45 gibi and when that transaction founds a duplicate project it dies even if we catch the exception from sqla
16:15:48 sean-k-mooney i wonder if we shoudl drop those tables in the future and just store the uuids direcly in the allcoation table
16:15:51 dansmith gibi: further explains the large-ness of the transaction being a problem I guess
16:16:00 bauzas dansmith: that's why I nearly consider this as a regression
16:16:13 dansmith bauzas: definitely sounds like a regression
16:16:19 bauzas we stepped into this mess in Xena
16:16:39 opendevreview Merged openstack/nova master: Parse extended resource request from the port data https://review.opendev.org/c/openstack/nova/+/800085
16:16:39 dansmith sean-k-mooney: only if we want it to be slower and more bloated I think :)
16:17:15 gibi the reason this become a single transaction is that consumer_type can be updated during PUT /allocations and if that happens in a separate transaction then we leak the update
16:17:17 sean-k-mooney well right now we are going to have to do a select by uuid to get the id in the project/user tabel then join based on that to the allcoation
16:17:39 bauzas I have no idea because I haven't looked at code yet, but is it crazy to switch back to the 2-step transactional model we had before ?
16:17:50 dansmith gibi: but we're just recording projects for relational purposes, so even if the allocation put doesn't succeed, it's fine that we've created the project record right?
16:17:51 sean-k-mooney so im not sure it would be slow to just to a select againt the allcoation table directly with an index on the user_id/project_id columns
16:18:00 bauzas ie. get the projet or create it, first, then do the allocation update
16:18:10 gibi dansmith: yes, for project and user it is fine, for consumer_type it is not fine
16:18:14 bauzas into two transactions
16:18:16 dansmith gibi: right
16:19:03 gibi so we can move back to many transactions but then we need to figure out how to move the consumer_type update to the last big transaction
16:19:19 dansmith yeah, which is surgery for sure
16:19:36 dansmith was melwitt the one that wrote the consumer type patches?
16:19:40 dansmith maybe we should get her consult
16:19:51 gibi it was melwitt who updated it after cdent left
16:20:03 dansmith well, right
16:20:06 gibi so she knows more
16:20:16 dansmith $current_owner = 'melwitt' is what I meant :)
16:20:20 gibi yeah
16:20:44 gibi OK, I guess that is it for now about this bug I will ping melwitt about it and record this discussion inthe bug
16:20:48 gibi moving on
16:21:03 gibi is there any other nova bug (gate or not) we need to discuss?
16:21:20 bauzas did we put the hot potato into someone's hand who isn't there ? excellent outcome of this discussion :p
16:21:33 opendevreview Merged openstack/nova master: Transfer RequestLevelParams from ports to scheduling https://review.opendev.org/c/openstack/nova/+/791506
16:21:39 gibi bauzas: I can still feel the responsibility until I can talk to melwitt
16:21:40 dansmith wasn't it in her hands to begin with? I think it's okay :P
16:21:58 gibi :D
16:22:10 gibi I see there is no other bug to talk about great :D
16:22:12 bauzas moving on
16:22:21 gibi Placement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly
16:22:25 gibi they are greeen
16:22:44 gibi and last but not least be prepared that the gate will be slow this week due to Milestone 3
16:23:03 gibi which brings us to
16:23:03 gibi #topic Release Planning
16:23:08 gibi Release tracking etherpad #link https://etherpad.opendev.org/p/nova-xena-rc-potential
16:23:16 gibi Feature freeze is 3rd of Sept which is end of this week.
16:23:20 gibi Also we have Milestone 3 end of this week that marks the last release of our client libs. Release patches:

Earlier   Later