Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-28
16:04:01 jmlowe mriedem: versioned notifications for ceilometer are still tagged as a wishlist bug
16:06:55 mriedem jmlowe: umm
16:07:06 mriedem if you're using unversioned, there should be no difference
16:07:23 Sundar jaypipes: I found the source for fpgaconf.c : https://github.com/OPAE/opae-sdk/tree/master/tools/base/fpgaconf
16:07:24 jmlowe mriedem: that was learned the hard way
16:07:40 mriedem jmlowe: can you expand? did you upgrade and the format option was using 'both'?
16:08:48 mriedem i remember godaddy saying in boston that the instance.exists notifications in particular hammer their MQ
16:09:59 sean-k-mooney Sundar: could we do a poc usign a FakeDriver the simulates a fake device and uses that to test this end to end and there for remove the hardware depency
16:10:26 sean-k-mooney we have asked this in the past
16:10:45 sean-k-mooney it woudl be useful for ci if nothing else
16:10:48 jmlowe Now on queens, upgraded in place starting from liberty, didn't really pay attention, just assumed everything would work, the versioned notifications queue grew without bound until I finally got smart an capped nova with legacy notifications
16:11:57 mriedem ok, so it was sending both then
16:12:07 jmlowe whatever the default is
16:12:11 mriedem it's 'both'
16:12:18 mriedem gibi: i almost wonder if we should change the default to unversioned...
16:12:41 mriedem i know we want people to use versioned, but there aren't any projects working on that
16:12:49 mriedem or,
16:13:05 mriedem maybe we should start a 'performance / scale considerations' doc in nova for stuff like this
16:13:28 mriedem "rabbit got you down? check your notifications settings."
16:14:35 gibi mriedem: does changing the default in a bugfix is safe from config compatibility perspective?
16:15:07 jmlowe That may thwart my ambitions to be a obscenely high paid consultant
16:15:19 bauzas mriedem: artom: there ? I have a concern on https://review.openstack.org/#/c/599587/
16:15:32 bauzas most of my comments are nits but the last one is important
16:15:46 bauzas artom: mriedem: https://review.openstack.org/#/c/599587/8/specs/stein/approved/numa-aware-live-migration.rst@327
16:16:13 bauzas the question is : once we implement this, we will change how we will accept live migrations
16:16:28 bauzas should we signal it ?
16:16:52 sean-k-mooney bauzas: how do you mean
16:16:54 bauzas or just drop a release note saying "by Stein, NUMA-aware live migrations will be unaccepted"
16:17:01 artom bauzas, we could make it configurable, sort like what stephenfin is proposing with https://review.openstack.org/#/c/611088/
16:17:09 mriedem didn't this already come up in https://review.openstack.org/#/c/611088/
16:17:10 mriedem yeah
16:17:27 artom bauzas, but... why? Why would anyone want to live migrate in the middle of an upgrade with mixed computes?
16:17:42 artom Live migrate *knowing* that things will mostly likely go south
16:18:05 sean-k-mooney artom: it not that uncommon
16:18:17 bauzas artom: it's a possibility yes
16:18:23 bauzas that's even a common pattern
16:18:31 sean-k-mooney in that case they are likely specifyign the host as they are freeing up old host to upgrade
16:18:37 bauzas operators do a lot of migrations when they upgrade
16:18:48 dansmith artom: live migration during an upgrade is the primary mechanism that most people use
16:19:07 dansmith artom: *lots* of people refuse to upgrade a compute node until they've moved everything off of it
16:19:14 bauzas tbh, I'm fine with dropping a release note saying there will be an impact
16:19:21 artom dansmith, fair enough
16:19:49 bauzas given worloads should move from old to new
16:19:56 bauzas and not from new to old
16:19:59 sean-k-mooney bauzas: we could do what we did for multiple port bindings and just fall back to the old behavior
16:20:12 bauzas oh shit, the problem is with old to new
16:20:17 bauzas not the contrary, my bad
16:20:41 artom I just don't see how we can realistically handle that
16:20:44 bauzas so, yeah, we could shoot operators in the foot
16:20:58 bauzas artom: we could make a flag during the upgrade
16:21:03 artom Keep the old code paths intact while somehow adding claims and all the new XML stuff
16:21:13 bauzas and say 'if you need so, it's on you, folks"
16:21:21 sean-k-mooney artom: for old to new we can fallback to the old behvior
16:21:38 bauzas new to old isn't a problem
16:21:41 bauzas new to new isn't a problem
16:21:42 artom sean-k-mooney, can we? How do we even implement that?
16:21:45 bauzas old to new is the problem
16:22:00 sean-k-mooney e.g. claim nothing and tell people to migrate again after migrate to fix everything
16:22:02 bauzas so yeah, we somehow need to keep the broken behaviour
16:22:05 artom It's going to be way ugly with a whole bunch of conditionals all over the place
16:22:15 bauzas artom: I know, it's freaking ugly
16:22:46 bauzas but I don't see operators be happy with what you propose :p
16:22:46 artom Unless we do the new stuff in entirely new RPC calls/casts
16:22:55 gibi mriedem, jmlowe: I reported https://bugs.launchpad.net/nova/+bug/1805659
16:22:55 openstack Launchpad bug 1805659 in OpenStack Compute (nova) "nova notifications hammering the message bus" [Undecided,New]
16:22:58 sean-k-mooney artom: its what we did for multiple port bindings. we check if a field exitsing in the migrate data if not we do the old stuff if its there we do the new way
16:23:04 artom bauzas, yeah, probably true
16:23:05 dansmith artom: that's how we do things gracefully
16:23:14 bauzas couldn't you drop the move claim if the RPC argument is not there ?
16:23:18 dansmith artom: i.e. pass a flag, and if it's missing, assume it's an old compute and do the old thing
16:23:19 artom dansmith, what is? New RPC version?
16:23:29 dansmith you can't see the rpc version on the receiving end
16:23:32 dansmith you need a flag
16:23:32 bauzas what dansmith said and me :)
16:23:36 dansmith do_the_new_thing=True
16:23:46 bauzas that's what we generally do
16:23:49 artom dansmith, I don't disagree, but it's going to be a mess
16:23:58 bauzas artom: I can point you some code I wrote that does the signaling
16:24:18 dansmith artom: it's how you have to do it
16:24:19 dansmith it's how everything we do that involves old/new services works
16:24:30 bauzas artom: sure, but we somehow need to know that the live migration comes from an old compute hence us releasing the check
16:24:41 bauzas and not doing the claim
16:25:05 mriedem if you have an old dest compute, the migrate_data won't have the numa stuff from the claim right?
16:25:24 mriedem so the source (if new) can't rely on it
16:25:35 artom mriedem, other way around, new dest, old source
16:25:44 artom So source won't send the updated XML, but dest will have claimed for it
16:25:55 artom So we need to do conditional claiming based on source compute version
16:26:15 mriedem i think that's what the file-backed memory live migration does...
16:26:16 bauzas you could make the migrate_data parameter a sentinel
16:26:22 mriedem it checks the compute service version for the source from the dest
16:26:47 artom Alright, lemme update the spec
16:27:02 bauzas artom: just change this
16:27:06 bauzas other things were nits
16:27:16 bauzas just for documenting the spec
16:27:18 artom Alright if I keep it kinda high-level? So "conditional claim", but without specifying what that condition will look like (flag, version checking, etc)?
16:27:20 bauzas later when we review
16:27:28 dansmith artom: you can't check the version
16:27:42 mriedem artom: this is the code i'm thinking of https://github.com/openstack/nova/blob/62245235bc15da6abcdfd3df1c24bd856d69fbb4/nova/virt/libvirt/driver.py#L6636
16:27:51 artom dansmith, I guess not on the compute, eh?
16:27:59 artom Only conductor, so it has to be flag

Earlier   Later