Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-10
20:25:00 zzzeek AND, if the latter, what if the two UPDATE statements are changing "x" to the *same* value? this is ultimately a nova question
20:25:28 zzzeek b.c. the issue is observed in the instance shelving logic
20:35:21 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656
20:35:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269
20:35:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478
20:36:43 mriedem zzzeek: a la this bug? https://bugs.launchpad.net/nova/+bug/1821373
20:36:44 openstack Launchpad bug 1821373 in OpenStack Compute (nova) "Most instance actions can be called concurrently" [Undecided,New]
20:37:00 zzzeek mriedem: yes
20:37:23 zzzeek mriedem: mdbooth_ has updated me that his proposed solution won't work
20:39:22 mriedem i think he mentioned something to that effect in irc awhile back for this bug but i don't remember what or why
20:44:06 mriedem gotta run
20:48:57 cdent so many things
20:51:28 cdent zzzeek: since you're sort of around, I wanted to warn you that at some point we might be hassling you for some advice on some performance improvements in placement.
20:51:47 cdent but not now. later.
20:51:54 zzzeek cdent: that should be good since placement looks to be straightforward
20:52:03 cdent it _was_
20:52:53 cdent but the nested stuff being added this cycle are going to be mind bending
20:55:10 zzzeek cdent: what kind of nesting
21:07:09 edleafe zzzeek: representing compute nodes containing NUMA nodes containing memory, for example
21:07:18 cdent zzzeek: sorry was away. the nesting is the "nested providers" concept. The new features are described (in brief) at https://storyboard.openstack.org/#!/story/2005575
21:07:54 cdent zzzeek edleafe has done some interesting work to switching a graph database (which is a better fit for some of these things) but we're in the standard position of "we've got this other stuff already"
21:09:00 zzzeek cdent: so for now are you looking at...adjacency list schema ?
21:10:11 cdent zzzeek: no explicit decisions yet. the linked email thread has some ideas. But the lack of decisions is why I was saying we'll probably want some input, but not quite yet. It's more of a heads up for a later discussion rather than a now discussion
21:11:25 cdent on top of that: solutions for the existing features are going to need some work to scale into the 100s of thousands of resource providers
21:11:42 cdent but we need to do some accurate measuring first
21:19:08 zzzeek edleafe is dying to get off SQL :P
21:19:09 zzzeek :)
21:19:31 edleafe zzzeek: Just for this case.
21:19:43 zzzeek edleafe: yeah graph DBs are awesome
21:19:53 zzzeek edleafe: huge crazy new dependencies not as much :)
21:20:04 edleafe zzzeek: I got nested providers working in a couple of days at the summit last week
21:20:06 zzzeek edleafe: might be one of htose cases where you have multiple backends
21:20:26 edleafe I just use Neo4j in docker :)
21:22:14 zzzeek edleafe: we still have to have RPMs that build it out, we have memory reuirements that java VM adds some weight towards, etc
21:22:24 zzzeek edleafe: by "we" I mean red hat
21:23:55 edleafe zzzeek: Sure, I understand all that. I just want to show it working, and solving the problems that we've spent years trying to solve using SQL. If that gets done and enough people feel that this is the way to go, then we can worry about what is needed for build/deploy
21:24:37 zzzeek edleafe: for small graphs, I use adjacency list. if you need a huge deep graph every time, then that won't work
21:25:18 jaypipes zzzeek: apologies, having net issues... reading back
21:25:46 cdent I think our model is many trees, each <= 7 levels deep, not very broad
21:26:16 edleafe cdent: for nested, agree. Shared providers is the opposite
21:27:14 cdent depends on how we model the shared association. in a graph db, yes
21:28:23 cdent I tend to like shared as group, not tree
21:35:16 openstackgerrit Eric Fried proposed openstack/os-resource-classes master: Propose ACCELERATOR_{FPGA|GPU} resource classes https://review.opendev.org/657464
21:36:17 edleafe cdent: I was just commenting on the trees reference. In the graph, sharing is just another relation. With Many:1 sharing, it kind of looks like a flower, not a tree: https://bit.ly/2VUI52U
21:38:30 jaypipes zzzeek: set global wsrep_causal_reads=1; <-- do that if you want synchronous replication behaviour.
21:38:59 zzzeek jaypipes: OK so you can confirm that comapre and swap can fail for multimaster if that's not set ?
21:39:25 zzzeek jaypipes: this might be what's needed for https://bugs.launchpad.net/nova/+bug/1821373
21:39:26 openstack Launchpad bug 1821373 in OpenStack Compute (nova) "Most instance actions can be called concurrently" [Undecided,New]
21:43:35 jaypipes ? + 1 WHERE generation = ? AND uuid = ?
21:43:35 jaypipes zzzeek: I'm not 100% sure, but I believe so. the issue, however, is that the compare-and-swap technique should be done to increment a field and check that field value is at a previous read-view in the WHERE clause. in other words, doing things like UPDATE instances SET status = 'active' WHERE status IN (<list of statuses>) AND instances.uuid = ? is inherently not as safe/efficient/retryable-with-confidence as UPDATE instances SET generation =
21:44:22 jaypipes zzzeek: that's the fundamental problem IMHO with the whole "check my instance status is in these list of states and set the status to X" checks that nova does.
21:45:12 zzzeek jaypipes: if the UPDATE included a unique version or timestamp of some kind does that help? if galera sees two UPDATE statements setting it to a different value ?
21:48:41 jaypipes zzzeek: maybe? :) galera already sends around essentially a generation for the innodb records affected by a transaction writeset, AFAIK. I just think doing the "compare" part of the compare-and-swap functionality with a "loose match" like "status IN <....>" isn't as good as comparator that was specifically designed to inform the caller that "yes, someone else changed this record since you last read a view of it". hope that makes sense.
21:50:13 jaypipes zzzeek: this is why, in placement land, we always do the compare-and-swap using the `UPDATE tbl SET generation = ($LAST_READ_GENERATION + 1) WHERE pk = $PK AND generation = $LAST_READ_GENERATION` strategy
21:50:15 zzzeek jaypipes: the specific case in that issue we are looking for an exisitng status of NULL
21:51:41 jaypipes zzzeek: yeah, but AFAIK, the code you're describing isn't really a compare-and-swap. it's more just a "hey, check that I'm not, say, in the process of deleting this instance when I try to unshelve it"
21:51:50 zzzeek jaypipes: well yes I was saying, it's more reliable if we are setting it to a new value, however, if two transactions on different nodes hit it at the same time they will see the same LAST_READ_GENERATION value
21:52:25 jaypipes zzzeek: yes, they will.
21:52:38 jaypipes zzzeek: and if both attempt to update, one will fail of course.
21:53:30 zzzeek jaypipes: one fails because of the SET clasue specifically ?
21:54:17 jaypipes zzzeek: no, one will fail because WHERE generation = $LAST_READ_GENERATION will fail.
21:54:22 jaypipes to return a row.
21:54:55 jaypipes zzzeek: so, the SQL won't fail, per-se, it's just the transaction will return 0 rows affected.
21:55:09 jaypipes zzzeek: which is the thing we look for to trigger a rollback of the entire transaction.
21:55:18 zzzeek jaypipes: but what if one UPDATE processes, sends out the writeset which includes the new value, however the other node gets an UPDATE, due to replcation latency it also sees the same value, also emits an UPDATE, no failure
21:55:28 zzzeek but updates the row
21:55:40 zzzeek beause the new vaule wasn't there yet b.c. no wsrep_causal_reads
21:56:05 zzzeek jaypipes: this gets into, I have no idea how the galera writeset certification works
21:56:32 zzzeek jaypipes: I would think that certifiaction should be, transaction modified this row, this other transacvtion is modifying the same generation of that row, so it fails
21:56:39 zzzeek which means this issue is non-existent
21:56:43 zzzeek e.g. mvcc generation
21:57:43 jaypipes that is essentially how it works, yes.
21:57:50 jaypipes https://github.com/openstack/placement/blob/master/placement/objects/resource_provider.py#L960
21:58:52 zzzeek jaypipes: OK. So, if the SHELVE thing is looking explicitly for NULL and changes to SHELVING as described in the launchpad, no failure ? how does it fail ?
21:58:56 jaypipes the wsrep_causal_reads is about reads only. writes that attempt to update the same record when a diff trx changed that record are never allowed.
21:59:07 zzzeek jaypipes: right that's what I sort of thought
21:59:26 jaypipes zzzeek: apooogies, I haven't read the bug report yet.
21:59:32 jaypipes lemme do that now. one minute.
22:00:40 jaypipes zzzeek: I would take issue with mdbooth_'s statement "This is intended to act as a robust gate against 2 instance actions happening concurrently." :)
22:00:48 jaypipes it's not a robust gate at all.
22:01:01 jaypipes it's a super coarse-grained check
22:01:15 zzzeek jaypipes: OK he is stumped on this and I dont really know the details of this system, I just wrote the UPDATE statement four years ago
22:01:23 zzzeek do you have anything you can add to that launchapd?
22:01:42 jaypipes in fact, it's not a gate at all. it's nothing more than a very simple sanity check that exists outside of any transactional context AFAIK
22:02:03 zzzeek jaypipes: really? it doesnt seem that way to me, I assume this is on enginefacade and there should be a tx
22:02:03 jaypipes zzzeek: yeah, I will add a note to it.
22:02:32 jaypipes zzzeek: one would assume that. I have no real way of verifying it's in the same trx though.
22:04:55 zzzeek thanks for the chat jaypipes
22:33:45 openstackgerrit Eric Fried proposed openstack/os-resource-classes master: Propose ACCELERATOR_{FPGA|GPU} resource classes https://review.opendev.org/657464
22:54:21 openstackgerrit Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.opendev.org/603955
23:22:12 openstackgerrit Merged openstack/nova master: Add ironic driver image type capabilities https://review.opendev.org/655729
23:22:18 openstackgerrit Merged openstack/nova master: Add vmware driver image type capabilities https://review.opendev.org/655730
#openstack-nova - 2019-05-11
02:55:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Trivial: Adds comments and tests for scheduler https://review.opendev.org/658516
04:02:14 openstackgerrit gugug proposed openstack/os-resource-classes master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658519
07:42:58 openstackgerrit gugug proposed openstack/nova-specs master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658528
08:43:10 openstackgerrit Tetsuro Nakamura proposed openstack/nova stable/rocky: Enable nested allocation candidates in scheduler https://review.opendev.org/658533
21:48:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648662
22:11:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269
22:11:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478
22:11:58 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656

Earlier   Later