Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-10
21:38:30 jaypipes zzzeek: set global wsrep_causal_reads=1; <-- do that if you want synchronous replication behaviour.
21:38:59 zzzeek jaypipes: OK so you can confirm that comapre and swap can fail for multimaster if that's not set ?
21:39:25 zzzeek jaypipes: this might be what's needed for https://bugs.launchpad.net/nova/+bug/1821373
21:39:26 openstack Launchpad bug 1821373 in OpenStack Compute (nova) "Most instance actions can be called concurrently" [Undecided,New]
21:43:35 jaypipes zzzeek: I'm not 100% sure, but I believe so. the issue, however, is that the compare-and-swap technique should be done to increment a field and check that field value is at a previous read-view in the WHERE clause. in other words, doing things like UPDATE instances SET status = 'active' WHERE status IN (<list of statuses>) AND instances.uuid = ? is inherently not as safe/efficient/retryable-with-confidence as UPDATE instances SET generation =
21:43:35 jaypipes ? + 1 WHERE generation = ? AND uuid = ?
21:44:22 jaypipes zzzeek: that's the fundamental problem IMHO with the whole "check my instance status is in these list of states and set the status to X" checks that nova does.
21:45:12 zzzeek jaypipes: if the UPDATE included a unique version or timestamp of some kind does that help? if galera sees two UPDATE statements setting it to a different value ?
21:48:41 jaypipes zzzeek: maybe? :) galera already sends around essentially a generation for the innodb records affected by a transaction writeset, AFAIK. I just think doing the "compare" part of the compare-and-swap functionality with a "loose match" like "status IN <....>" isn't as good as comparator that was specifically designed to inform the caller that "yes, someone else changed this record since you last read a view of it". hope that makes sense.
21:50:13 jaypipes zzzeek: this is why, in placement land, we always do the compare-and-swap using the `UPDATE tbl SET generation = ($LAST_READ_GENERATION + 1) WHERE pk = $PK AND generation = $LAST_READ_GENERATION` strategy
21:50:15 zzzeek jaypipes: the specific case in that issue we are looking for an exisitng status of NULL
21:51:41 jaypipes zzzeek: yeah, but AFAIK, the code you're describing isn't really a compare-and-swap. it's more just a "hey, check that I'm not, say, in the process of deleting this instance when I try to unshelve it"
21:51:50 zzzeek jaypipes: well yes I was saying, it's more reliable if we are setting it to a new value, however, if two transactions on different nodes hit it at the same time they will see the same LAST_READ_GENERATION value
21:52:25 jaypipes zzzeek: yes, they will.
21:52:38 jaypipes zzzeek: and if both attempt to update, one will fail of course.
21:53:30 zzzeek jaypipes: one fails because of the SET clasue specifically ?
21:54:17 jaypipes zzzeek: no, one will fail because WHERE generation = $LAST_READ_GENERATION will fail.
21:54:22 jaypipes to return a row.
21:54:55 jaypipes zzzeek: so, the SQL won't fail, per-se, it's just the transaction will return 0 rows affected.
21:55:09 jaypipes zzzeek: which is the thing we look for to trigger a rollback of the entire transaction.
21:55:18 zzzeek jaypipes: but what if one UPDATE processes, sends out the writeset which includes the new value, however the other node gets an UPDATE, due to replcation latency it also sees the same value, also emits an UPDATE, no failure
21:55:28 zzzeek but updates the row
21:55:40 zzzeek beause the new vaule wasn't there yet b.c. no wsrep_causal_reads
21:56:05 zzzeek jaypipes: this gets into, I have no idea how the galera writeset certification works
21:56:32 zzzeek jaypipes: I would think that certifiaction should be, transaction modified this row, this other transacvtion is modifying the same generation of that row, so it fails
21:56:39 zzzeek which means this issue is non-existent
21:56:43 zzzeek e.g. mvcc generation
21:57:43 jaypipes that is essentially how it works, yes.
21:57:50 jaypipes https://github.com/openstack/placement/blob/master/placement/objects/resource_provider.py#L960
21:58:52 zzzeek jaypipes: OK. So, if the SHELVE thing is looking explicitly for NULL and changes to SHELVING as described in the launchpad, no failure ? how does it fail ?
21:58:56 jaypipes the wsrep_causal_reads is about reads only. writes that attempt to update the same record when a diff trx changed that record are never allowed.
21:59:07 zzzeek jaypipes: right that's what I sort of thought
21:59:26 jaypipes zzzeek: apooogies, I haven't read the bug report yet.
21:59:32 jaypipes lemme do that now. one minute.
22:00:40 jaypipes zzzeek: I would take issue with mdbooth_'s statement "This is intended to act as a robust gate against 2 instance actions happening concurrently." :)
22:00:48 jaypipes it's not a robust gate at all.
22:01:01 jaypipes it's a super coarse-grained check
22:01:15 zzzeek jaypipes: OK he is stumped on this and I dont really know the details of this system, I just wrote the UPDATE statement four years ago
22:01:23 zzzeek do you have anything you can add to that launchapd?
22:01:42 jaypipes in fact, it's not a gate at all. it's nothing more than a very simple sanity check that exists outside of any transactional context AFAIK
22:02:03 jaypipes zzzeek: yeah, I will add a note to it.
22:02:03 zzzeek jaypipes: really? it doesnt seem that way to me, I assume this is on enginefacade and there should be a tx
22:02:32 jaypipes zzzeek: one would assume that. I have no real way of verifying it's in the same trx though.
22:04:55 zzzeek thanks for the chat jaypipes
22:33:45 openstackgerrit Eric Fried proposed openstack/os-resource-classes master: Propose ACCELERATOR_{FPGA|GPU} resource classes https://review.opendev.org/657464
22:54:21 openstackgerrit Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.opendev.org/603955
23:22:12 openstackgerrit Merged openstack/nova master: Add ironic driver image type capabilities https://review.opendev.org/655729
23:22:18 openstackgerrit Merged openstack/nova master: Add vmware driver image type capabilities https://review.opendev.org/655730
#openstack-nova - 2019-05-11
02:55:28 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Trivial: Adds comments and tests for scheduler https://review.opendev.org/658516
04:02:14 openstackgerrit gugug proposed openstack/os-resource-classes master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658519
07:42:58 openstackgerrit gugug proposed openstack/nova-specs master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658528
08:43:10 openstackgerrit Tetsuro Nakamura proposed openstack/nova stable/rocky: Enable nested allocation candidates in scheduler https://review.opendev.org/658533
21:48:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648662
22:11:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478
22:11:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269
22:11:58 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656
#openstack-nova - 2019-05-12
00:48:15 openstackgerrit Tetsuro Nakamura proposed openstack/nova stable/rocky: Enable nested allocation candidates in scheduler https://review.opendev.org/658533
02:05:20 openstackgerrit Merged openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648662
03:06:24 openstackgerrit Merged openstack/nova master: Add xenapi driver image type capabilities https://review.opendev.org/655731
#openstack-nova - 2019-05-13
00:52:30 openstackgerrit zhufl proposed openstack/nova master: Change some URLs to point to better targets https://review.opendev.org/658312
02:31:43 openstackgerrit Yongli He proposed openstack/nova-specs master: add spec "show-server-numa-topology" https://review.opendev.org/658716
04:05:16 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.opendev.org/575311
04:05:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.opendev.org/575581
04:05:51 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.opendev.org/576017
04:06:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.opendev.org/576018
04:06:33 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.opendev.org/576019
04:06:50 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020
04:07:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027
04:07:47 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031
04:08:06 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299
04:08:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344
04:08:44 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673
04:09:01 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.opendev.org/576676
04:09:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.opendev.org/576689
04:09:38 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.opendev.org/576709
04:09:57 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712
04:10:17 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.opendev.org/566153
04:10:31 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.opendev.org/637716
04:10:48 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.opendev.org/641200
04:11:10 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Fix a description for config_drive parameter https://review.opendev.org/653683
06:17:49 openstackgerrit Yongli He proposed openstack/nova master: Add server sub-resource topology API https://review.opendev.org/621476
06:38:19 openstackgerrit Merged openstack/nova master: Change some URLs to point to better targets https://review.opendev.org/658312
08:05:33 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/os-vif stable/queens: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655694
08:38:55 openstackgerrit Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.opendev.org/645520
09:24:24 ralonsoh stephenfin, hi. I'm having problems with https://review.opendev.org/#/c/655694/ (Queens) because of sphinx
09:24:41 ralonsoh stephenfin, is ok if I cherry-pick https://review.opendev.org/#/c/479862/ to Queens?
09:57:06 jangutter ralonsoh: hmmm, the other option is to pin sphinx to an older version for the queens release.
09:57:56 ralonsoh I'll take a look at sphinx versions to know which one supports both, 2 and 3
09:58:12 ralonsoh jangutter, probably <2.0
09:58:36 jangutter ralonsoh: yep, out of interest, what does Nova do for queens?
09:58:53 ralonsoh jangutter, I'll review this now
10:02:44 stephenfin ralonsoh: Sphinx isn't the problem. We need this 96b5a11d
10:03:07 stephenfin Sorry, https://opendev.org/openstack/os-vif/commit/96b5a11d
10:03:59 ralonsoh stephenfin, yes, I saw this. os-vif was not using upper-constraints
10:04:11 stephenfin yup
10:04:14 stephenfin bad os-vif
10:04:15 stephenfin bad
10:04:21 ralonsoh hehehe
10:04:38 ralonsoh should I cherry-pick this patch to previous versions? Queens?
10:05:05 stephenfin Yeah, but change the URL to https://releases.openstack.org/constraints/upper/queens

Earlier   Later