| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-28 | |||
| 15:59:08 | jmlowe | I have unversioned, was looking at rocky release notes and saw that instance exits has been transitioned and that's the one event I really need to consume with ceilometer/panko which doesn't do versioned yet | |
| 15:59:18 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Support volume-backed server rebuild https://review.openstack.org/532407 | |
| 15:59:41 | mriedem | jmlowe: just leave as unversioned then, | |
| 15:59:47 | mriedem | as gibi said, we haven't removed the unversioned notifications | |
| 16:00:04 | gibi | jmlowe: if https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format is configured to both or unversioned then nova still sends the old instance.exists | |
| 16:00:07 | mriedem | most, if not all, openstack projects that consume notifications from nova are still using unversoined notifications | |
| 16:01:07 | mriedem | dansmith: i'm +2 on the volume-backed rebuild spec now https://review.openstack.org/#/c/532407/ | |
| 16:02:36 | jmlowe | ok, perfect, thanks, needed a sanity check to make sure before I happily upgraded myself into a sisyphean custom patch cycle | |
| 16:03:09 | jmlowe | mriedem: my exploding rabbit queues say otherwise | |
| 16:04:01 | jmlowe | mriedem: versioned notifications for ceilometer are still tagged as a wishlist bug | |
| 16:06:55 | mriedem | jmlowe: umm | |
| 16:07:06 | mriedem | if you're using unversioned, there should be no difference | |
| 16:07:23 | Sundar | jaypipes: I found the source for fpgaconf.c : https://github.com/OPAE/opae-sdk/tree/master/tools/base/fpgaconf | |
| 16:07:24 | jmlowe | mriedem: that was learned the hard way | |
| 16:07:40 | mriedem | jmlowe: can you expand? did you upgrade and the format option was using 'both'? | |
| 16:08:48 | mriedem | i remember godaddy saying in boston that the instance.exists notifications in particular hammer their MQ | |
| 16:09:59 | sean-k-mooney | Sundar: could we do a poc usign a FakeDriver the simulates a fake device and uses that to test this end to end and there for remove the hardware depency | |
| 16:10:26 | sean-k-mooney | we have asked this in the past | |
| 16:10:45 | sean-k-mooney | it woudl be useful for ci if nothing else | |
| 16:10:48 | jmlowe | Now on queens, upgraded in place starting from liberty, didn't really pay attention, just assumed everything would work, the versioned notifications queue grew without bound until I finally got smart an capped nova with legacy notifications | |
| 16:11:57 | mriedem | ok, so it was sending both then | |
| 16:12:07 | jmlowe | whatever the default is | |
| 16:12:11 | mriedem | it's 'both' | |
| 16:12:18 | mriedem | gibi: i almost wonder if we should change the default to unversioned... | |
| 16:12:41 | mriedem | i know we want people to use versioned, but there aren't any projects working on that | |
| 16:12:49 | mriedem | or, | |
| 16:13:05 | mriedem | maybe we should start a 'performance / scale considerations' doc in nova for stuff like this | |
| 16:13:28 | mriedem | "rabbit got you down? check your notifications settings." | |
| 16:14:35 | gibi | mriedem: does changing the default in a bugfix is safe from config compatibility perspective? | |
| 16:15:07 | jmlowe | That may thwart my ambitions to be a obscenely high paid consultant | |
| 16:15:19 | bauzas | mriedem: artom: there ? I have a concern on https://review.openstack.org/#/c/599587/ | |
| 16:15:32 | bauzas | most of my comments are nits but the last one is important | |
| 16:15:46 | bauzas | artom: mriedem: https://review.openstack.org/#/c/599587/8/specs/stein/approved/numa-aware-live-migration.rst@327 | |
| 16:16:13 | bauzas | the question is : once we implement this, we will change how we will accept live migrations | |
| 16:16:28 | bauzas | should we signal it ? | |
| 16:16:52 | sean-k-mooney | bauzas: how do you mean | |
| 16:16:54 | bauzas | or just drop a release note saying "by Stein, NUMA-aware live migrations will be unaccepted" | |
| 16:17:01 | artom | bauzas, we could make it configurable, sort like what stephenfin is proposing with https://review.openstack.org/#/c/611088/ | |
| 16:17:09 | mriedem | didn't this already come up in https://review.openstack.org/#/c/611088/ | |
| 16:17:10 | mriedem | yeah | |
| 16:17:27 | artom | bauzas, but... why? Why would anyone want to live migrate in the middle of an upgrade with mixed computes? | |
| 16:17:42 | artom | Live migrate *knowing* that things will mostly likely go south | |
| 16:18:05 | sean-k-mooney | artom: it not that uncommon | |
| 16:18:17 | bauzas | artom: it's a possibility yes | |
| 16:18:23 | bauzas | that's even a common pattern | |
| 16:18:31 | sean-k-mooney | in that case they are likely specifyign the host as they are freeing up old host to upgrade | |
| 16:18:37 | bauzas | operators do a lot of migrations when they upgrade | |
| 16:18:48 | dansmith | artom: live migration during an upgrade is the primary mechanism that most people use | |
| 16:19:07 | dansmith | artom: *lots* of people refuse to upgrade a compute node until they've moved everything off of it | |
| 16:19:14 | bauzas | tbh, I'm fine with dropping a release note saying there will be an impact | |
| 16:19:21 | artom | dansmith, fair enough | |
| 16:19:49 | bauzas | given worloads should move from old to new | |
| 16:19:56 | bauzas | and not from new to old | |
| 16:19:59 | sean-k-mooney | bauzas: we could do what we did for multiple port bindings and just fall back to the old behavior | |
| 16:20:12 | bauzas | oh shit, the problem is with old to new | |
| 16:20:17 | bauzas | not the contrary, my bad | |
| 16:20:41 | artom | I just don't see how we can realistically handle that | |
| 16:20:44 | bauzas | so, yeah, we could shoot operators in the foot | |
| 16:20:58 | bauzas | artom: we could make a flag during the upgrade | |
| 16:21:03 | artom | Keep the old code paths intact while somehow adding claims and all the new XML stuff | |
| 16:21:13 | bauzas | and say 'if you need so, it's on you, folks" | |
| 16:21:21 | sean-k-mooney | artom: for old to new we can fallback to the old behvior | |
| 16:21:38 | bauzas | new to old isn't a problem | |
| 16:21:41 | bauzas | new to new isn't a problem | |
| 16:21:42 | artom | sean-k-mooney, can we? How do we even implement that? | |
| 16:21:45 | bauzas | old to new is the problem | |
| 16:22:00 | sean-k-mooney | e.g. claim nothing and tell people to migrate again after migrate to fix everything | |
| 16:22:02 | bauzas | so yeah, we somehow need to keep the broken behaviour | |
| 16:22:05 | artom | It's going to be way ugly with a whole bunch of conditionals all over the place | |
| 16:22:15 | bauzas | artom: I know, it's freaking ugly | |
| 16:22:46 | artom | Unless we do the new stuff in entirely new RPC calls/casts | |
| 16:22:46 | bauzas | but I don't see operators be happy with what you propose :p | |
| 16:22:55 | openstack | Launchpad bug 1805659 in OpenStack Compute (nova) "nova notifications hammering the message bus" [Undecided,New] | |
| 16:22:55 | gibi | mriedem, jmlowe: I reported https://bugs.launchpad.net/nova/+bug/1805659 | |
| 16:22:58 | sean-k-mooney | artom: its what we did for multiple port bindings. we check if a field exitsing in the migrate data if not we do the old stuff if its there we do the new way | |
| 16:23:04 | artom | bauzas, yeah, probably true | |
| 16:23:05 | dansmith | artom: that's how we do things gracefully | |
| 16:23:14 | bauzas | couldn't you drop the move claim if the RPC argument is not there ? | |
| 16:23:18 | dansmith | artom: i.e. pass a flag, and if it's missing, assume it's an old compute and do the old thing | |
| 16:23:19 | artom | dansmith, what is? New RPC version? | |
| 16:23:29 | dansmith | you can't see the rpc version on the receiving end | |
| 16:23:32 | bauzas | what dansmith said and me :) | |
| 16:23:32 | dansmith | you need a flag | |
| 16:23:36 | dansmith | do_the_new_thing=True | |
| 16:23:46 | bauzas | that's what we generally do | |
| 16:23:49 | artom | dansmith, I don't disagree, but it's going to be a mess | |
| 16:23:58 | bauzas | artom: I can point you some code I wrote that does the signaling | |
| 16:24:18 | dansmith | artom: it's how you have to do it | |
| 16:24:19 | dansmith | it's how everything we do that involves old/new services works | |
| 16:24:30 | bauzas | artom: sure, but we somehow need to know that the live migration comes from an old compute hence us releasing the check | |
| 16:24:41 | bauzas | and not doing the claim | |
| 16:25:05 | mriedem | if you have an old dest compute, the migrate_data won't have the numa stuff from the claim right? | |
| 16:25:24 | mriedem | so the source (if new) can't rely on it | |
| 16:25:35 | artom | mriedem, other way around, new dest, old source | |
| 16:25:44 | artom | So source won't send the updated XML, but dest will have claimed for it | |
| 16:25:55 | artom | So we need to do conditional claiming based on source compute version | |
| 16:26:15 | mriedem | i think that's what the file-backed memory live migration does... | |
| 16:26:16 | bauzas | you could make the migrate_data parameter a sentinel | |
| 16:26:22 | mriedem | it checks the compute service version for the source from the dest | |
| 16:26:47 | artom | Alright, lemme update the spec | |