| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-10 | |||
| 18:20:49 | mriedem | not really, but https://github.com/openstack/nova/blob/0cb1544106346664b4a53114458417ea62474b8c/nova/compute/api.py#L4853 | |
| 18:20:54 | mriedem | i wouldn't really expect it to either | |
| 18:20:57 | mriedem | nothing has needed it yet | |
| 18:21:15 | mriedem | https://github.com/openstack/nova/blob/0cb1544106346664b4a53114458417ea62474b8c/nova/compute/api.py#L4869 | |
| 18:21:23 | dansmith | right, but I was thinking we had discussed this before, | |
| 18:21:32 | mriedem | ah yes | |
| 18:21:33 | mriedem | Consequently we can currently assume that the context for # both the source and destination hosts of a migration is the # same. | |
| 18:21:33 | dansmith | and putting a comment like that seems like something you would have made me do :) | |
| 18:21:35 | mriedem | didn't read far enough | |
| 18:21:39 | mriedem | it was mdbooth | |
| 18:22:11 | openstackgerrit | Merged openstack/os-traits master: Update SEV trait docs to avoid misleading people https://review.opendev.org/655671 | |
| 18:22:53 | mriedem | so if the migration record has cross_cell_move=true i'll have to tell it to pull the dest host mapping via the host mapping | |
| 18:23:06 | mriedem | so still optimized for the non-cross-cell case | |
| 18:23:55 | dansmith | cool | |
| 18:35:19 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648662 | |
| 18:50:56 | eandersson | lol mriedem | |
| 18:54:33 | zzzeek | jaypipes: ping | |
| 18:56:08 | mriedem | hmm my genius test must not be so great http://paste.openstack.org/show/751238/ | |
| 18:56:11 | mriedem | since it should fail, but doesn't | |
| 18:56:26 | mriedem | was hoping to avoid stubbing things out in the functional test to test the multi-cell event routing | |
| 18:57:45 | mriedem | dansmith: is there maybe something in our rpc fixture stuff that wouldn't make this easy to test in a functional test because we're using fake rpc? | |
| 18:58:16 | dansmith | mriedem: er, I wouldn't think so.. you're using the multi cell fixture? | |
| 18:58:31 | mriedem | yeah, my functional tests work fine for multi-cell | |
| 18:58:49 | mriedem | could be my context manager yield pattern in ^ isn't actually asserting anything | |
| 19:04:00 | mriedem | no that routing seems to be working at least on one host | |
| 19:04:01 | mriedem | b'2019-05-10 15:02:18,969 DEBUG [nova.compute.manager] Processing event network-vif-plugged-60c6c823-ed75-46a0-a0c6-30aadb90e78c' | |
| 19:04:01 | mriedem | b'2019-05-10 15:02:18,970 DEBUG [nova.compute.manager] Received event network-vif-plugged-60c6c823-ed75-46a0-a0c6-30aadb90e78c' | |
| 19:28:11 | mriedem | i think it works in functional testing because we're not using multiple rpc transports https://review.opendev.org/#/c/396417/ | |
| 19:32:32 | mriedem | so i'm guessing in a multi-cell test we'd need to use per-cell rpc fixtures? | |
| 19:39:26 | efried | mriedem, tssurya: I'm +2 on https://review.opendev.org/#/c/648662/ - leaving for mriedem to +W. | |
| 19:45:04 | mriedem | oh the pressure is on | |
| 19:47:23 | ganso | mriedem: hey Matt. I am looking at https://review.opendev.org/#/c/658136/ ... it seems it needs https://github.com/openstack/nova/commit/94e620e87cb9349f799007f418ce94978bc33be1 | |
| 19:47:42 | ganso | mriedem: Rocky also doesn't have the methods assertFlavorMatchesUsage and assertRequestMatchesUsage | |
| 19:48:05 | ganso | mriedem: do you think it makes sense to backport the refactor? | |
| 19:48:44 | mriedem | no | |
| 19:48:55 | mriedem | there should be assertion methods you can use in queens | |
| 19:51:06 | ganso | mriedem: only assertFlavorMatchesAllocation | |
| 19:51:13 | ganso | mriedem: so the solution would be to cut down on some asserts | |
| 19:57:05 | mriedem | without looking at the test patch, you could rig up your own usage assertion using https://github.com/openstack/nova/blob/stable/queens/nova/tests/functional/test_servers.py#L1514 | |
| 19:57:17 | mriedem | not sure if you're asserting provider usage or consumer usage | |
| 19:57:39 | mriedem | i'm assuming the former | |
| 19:57:58 | ganso | mriedem: yup, doing that now. Basically re-coding the test to perform checks as it was done in queens | |
| 19:58:06 | ganso | mriedem: yea provider usage | |
| 20:13:21 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/queens: [DEBUG] Add functional confirm_migration_error test https://review.opendev.org/658136 | |
| 20:22:17 | jaypipes | zzzeek: pong | |
| 20:22:24 | zzzeek | hey | |
| 20:22:28 | jaypipes | heya :) | |
| 20:22:55 | zzzeek | jaypipes: time warp back to http://www.joinfu.com/2015/01/understanding-reservations-concurrency-locking-in-nova/ | |
| 20:24:02 | zzzeek | so given that galera can have a little bit of read latency: https://www.percona.com/blog/2013/03/03/investigating-replication-latency-in-percona-xtradb-cluster/ can that impact an UPDATE that says, "UPDATE ... SET x='bar' WHERE x = 'foo'" ? if two transactions both update, but write latency means transaction 2 still sees "foo" ? | |
| 20:24:34 | zzzeek | OR, do the UPDATE statements in a write-set get replayed such that they are serialized and it will detect this ? | |
| 20:25:00 | zzzeek | AND, if the latter, what if the two UPDATE statements are changing "x" to the *same* value? this is ultimately a nova question | |
| 20:25:28 | zzzeek | b.c. the issue is observed in the instance shelving logic | |
| 20:35:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269 | |
| 20:35:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656 | |
| 20:35:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478 | |
| 20:36:43 | mriedem | zzzeek: a la this bug? https://bugs.launchpad.net/nova/+bug/1821373 | |
| 20:36:44 | openstack | Launchpad bug 1821373 in OpenStack Compute (nova) "Most instance actions can be called concurrently" [Undecided,New] | |
| 20:37:00 | zzzeek | mriedem: yes | |
| 20:37:23 | zzzeek | mriedem: mdbooth_ has updated me that his proposed solution won't work | |
| 20:39:22 | mriedem | i think he mentioned something to that effect in irc awhile back for this bug but i don't remember what or why | |
| 20:44:06 | mriedem | gotta run | |
| 20:48:57 | cdent | so many things | |
| 20:51:28 | cdent | zzzeek: since you're sort of around, I wanted to warn you that at some point we might be hassling you for some advice on some performance improvements in placement. | |
| 20:51:47 | cdent | but not now. later. | |
| 20:51:54 | zzzeek | cdent: that should be good since placement looks to be straightforward | |
| 20:52:03 | cdent | it _was_ | |
| 20:52:53 | cdent | but the nested stuff being added this cycle are going to be mind bending | |
| 20:55:10 | zzzeek | cdent: what kind of nesting | |
| 21:07:09 | edleafe | zzzeek: representing compute nodes containing NUMA nodes containing memory, for example | |
| 21:07:18 | cdent | zzzeek: sorry was away. the nesting is the "nested providers" concept. The new features are described (in brief) at https://storyboard.openstack.org/#!/story/2005575 | |
| 21:07:54 | cdent | zzzeek edleafe has done some interesting work to switching a graph database (which is a better fit for some of these things) but we're in the standard position of "we've got this other stuff already" | |
| 21:09:00 | zzzeek | cdent: so for now are you looking at...adjacency list schema ? | |
| 21:10:11 | cdent | zzzeek: no explicit decisions yet. the linked email thread has some ideas. But the lack of decisions is why I was saying we'll probably want some input, but not quite yet. It's more of a heads up for a later discussion rather than a now discussion | |
| 21:11:25 | cdent | on top of that: solutions for the existing features are going to need some work to scale into the 100s of thousands of resource providers | |
| 21:11:42 | cdent | but we need to do some accurate measuring first | |
| 21:19:08 | zzzeek | edleafe is dying to get off SQL :P | |
| 21:19:09 | zzzeek | :) | |
| 21:19:31 | edleafe | zzzeek: Just for this case. | |
| 21:19:43 | zzzeek | edleafe: yeah graph DBs are awesome | |
| 21:19:53 | zzzeek | edleafe: huge crazy new dependencies not as much :) | |
| 21:20:04 | edleafe | zzzeek: I got nested providers working in a couple of days at the summit last week | |
| 21:20:06 | zzzeek | edleafe: might be one of htose cases where you have multiple backends | |
| 21:20:26 | edleafe | I just use Neo4j in docker :) | |
| 21:22:14 | zzzeek | edleafe: we still have to have RPMs that build it out, we have memory reuirements that java VM adds some weight towards, etc | |
| 21:22:24 | zzzeek | edleafe: by "we" I mean red hat | |
| 21:23:55 | edleafe | zzzeek: Sure, I understand all that. I just want to show it working, and solving the problems that we've spent years trying to solve using SQL. If that gets done and enough people feel that this is the way to go, then we can worry about what is needed for build/deploy | |
| 21:24:37 | zzzeek | edleafe: for small graphs, I use adjacency list. if you need a huge deep graph every time, then that won't work | |
| 21:25:18 | jaypipes | zzzeek: apologies, having net issues... reading back | |
| 21:25:46 | cdent | I think our model is many trees, each <= 7 levels deep, not very broad | |
| 21:26:16 | edleafe | cdent: for nested, agree. Shared providers is the opposite | |
| 21:27:14 | cdent | depends on how we model the shared association. in a graph db, yes | |
| 21:28:23 | cdent | I tend to like shared as group, not tree | |
| 21:35:16 | openstackgerrit | Eric Fried proposed openstack/os-resource-classes master: Propose ACCELERATOR_{FPGA|GPU} resource classes https://review.opendev.org/657464 | |
| 21:36:17 | edleafe | cdent: I was just commenting on the trees reference. In the graph, sharing is just another relation. With Many:1 sharing, it kind of looks like a flower, not a tree: https://bit.ly/2VUI52U | |
| 21:38:30 | jaypipes | zzzeek: set global wsrep_causal_reads=1; <-- do that if you want synchronous replication behaviour. | |
| 21:38:59 | zzzeek | jaypipes: OK so you can confirm that comapre and swap can fail for multimaster if that's not set ? | |
| 21:39:25 | zzzeek | jaypipes: this might be what's needed for https://bugs.launchpad.net/nova/+bug/1821373 | |
| 21:39:26 | openstack | Launchpad bug 1821373 in OpenStack Compute (nova) "Most instance actions can be called concurrently" [Undecided,New] | |
| 21:43:35 | jaypipes | zzzeek: I'm not 100% sure, but I believe so. the issue, however, is that the compare-and-swap technique should be done to increment a field and check that field value is at a previous read-view in the WHERE clause. in other words, doing things like UPDATE instances SET status = 'active' WHERE status IN (<list of statuses>) AND instances.uuid = ? is inherently not as safe/efficient/retryable-with-confidence as UPDATE instances SET generation = | |
| 21:43:35 | jaypipes | ? + 1 WHERE generation = ? AND uuid = ? | |