Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-28
15:46:47 jangutter we're falling into the trap of agreeing on stuff: which means it won't happen.
15:46:49 jaypipes Sundar: what's the likelihood of your PoC being available this year?
15:46:58 jaypipes Sundar: I understand you need permission from legal?
15:48:28 Sundar I'd like some agreement on the general direction of the POC. At the PTG, there was alignment on defining REST calls, and I wrote that up in a spec that is being reviewed. The discussion here seems more philosophical. I suggest that the POC should focus on clarifying the spec.
15:49:28 artom Sundar, I think at some point you'll have to accept that work might get wasted :(
15:49:41 dansmith Sundar: while everything is just words in a document, the discussion can get pretty philosophical pretty quick, as you have no doubt noticed
15:49:44 artom I've been observing this discussion from afar since... Dublin?
15:49:48 sean-k-mooney Sundar: can we do a simple poc of booting a vm with just a fixed fuction pci device
15:49:54 Sundar jaypipes: Yes, it requires legal clearance. I may get a requirement to follow a process, or likely multiple processes. I am hoping that, confining it to upstream code and no real devices will make it go faster
15:50:12 artom Bite the bullet, write some code, share it
15:50:23 artom If the code gets scrapped and needs to be rewritte, so be it
15:50:24 sean-k-mooney Sundar: the intel fpga stuff may but nova and cyborg working together dont
15:50:26 jaypipes Sundar: that's generally not what PoCs are, but ok :) My suggestion is to demo Nova actually being able to /usr/bin/fpgaconf some device. If you want to go through the trouble of getting all the REST API stuff done in order for Nova to shell out to /usr/bin/fpgaconf, go for it. :)
15:53:36 jmlowe If anybody remembers my slow placement api problem from a couple of days ago, it seems that NAT in the 4.15 kernel that ships with ubuntu 18.04 is problematic at scale
15:54:41 cdent jmlowe: I am a) relieved to hear that, b) sorry for your luck
15:55:06 cdent can you define "at scale"?
15:56:03 mriedem cern runs centos right?
15:56:05 mriedem tssurya: ^
15:56:32 jmlowe I have a quick question about versioned notifications, if a notification has been transitioned to versioned and the setting is for legacy does that mean the notification will never be emitted?
15:57:10 mriedem jmlowe: what is the value of this option https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format
15:57:13 mriedem in your nova.conf
15:57:15 jmlowe cdent: I was falling over at ~280, threshold is probably significantly lower, ubuntu 16.04 didn't cause problems
15:57:20 gibi jmlowe: if you configured nova to emit legacy notifications only then nova only emts the legacy ones
15:58:01 gibi jmlowe: we did not removed the legacy notifications, and we did not plan to remove them
15:59:08 jmlowe I have unversioned, was looking at rocky release notes and saw that instance exits has been transitioned and that's the one event I really need to consume with ceilometer/panko which doesn't do versioned yet
15:59:18 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Support volume-backed server rebuild https://review.openstack.org/532407
15:59:41 mriedem jmlowe: just leave as unversioned then,
15:59:47 mriedem as gibi said, we haven't removed the unversioned notifications
16:00:04 gibi jmlowe: if https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format is configured to both or unversioned then nova still sends the old instance.exists
16:00:07 mriedem most, if not all, openstack projects that consume notifications from nova are still using unversoined notifications
16:01:07 mriedem dansmith: i'm +2 on the volume-backed rebuild spec now https://review.openstack.org/#/c/532407/
16:02:36 jmlowe ok, perfect, thanks, needed a sanity check to make sure before I happily upgraded myself into a sisyphean custom patch cycle
16:03:09 jmlowe mriedem: my exploding rabbit queues say otherwise
16:04:01 jmlowe mriedem: versioned notifications for ceilometer are still tagged as a wishlist bug
16:06:55 mriedem jmlowe: umm
16:07:06 mriedem if you're using unversioned, there should be no difference
16:07:23 Sundar jaypipes: I found the source for fpgaconf.c : https://github.com/OPAE/opae-sdk/tree/master/tools/base/fpgaconf
16:07:24 jmlowe mriedem: that was learned the hard way
16:07:40 mriedem jmlowe: can you expand? did you upgrade and the format option was using 'both'?
16:08:48 mriedem i remember godaddy saying in boston that the instance.exists notifications in particular hammer their MQ
16:09:59 sean-k-mooney Sundar: could we do a poc usign a FakeDriver the simulates a fake device and uses that to test this end to end and there for remove the hardware depency
16:10:26 sean-k-mooney we have asked this in the past
16:10:45 sean-k-mooney it woudl be useful for ci if nothing else
16:10:48 jmlowe Now on queens, upgraded in place starting from liberty, didn't really pay attention, just assumed everything would work, the versioned notifications queue grew without bound until I finally got smart an capped nova with legacy notifications
16:11:57 mriedem ok, so it was sending both then
16:12:07 jmlowe whatever the default is
16:12:11 mriedem it's 'both'
16:12:18 mriedem gibi: i almost wonder if we should change the default to unversioned...
16:12:41 mriedem i know we want people to use versioned, but there aren't any projects working on that
16:12:49 mriedem or,
16:13:05 mriedem maybe we should start a 'performance / scale considerations' doc in nova for stuff like this
16:13:28 mriedem "rabbit got you down? check your notifications settings."
16:14:35 gibi mriedem: does changing the default in a bugfix is safe from config compatibility perspective?
16:15:07 jmlowe That may thwart my ambitions to be a obscenely high paid consultant
16:15:19 bauzas mriedem: artom: there ? I have a concern on https://review.openstack.org/#/c/599587/
16:15:32 bauzas most of my comments are nits but the last one is important
16:15:46 bauzas artom: mriedem: https://review.openstack.org/#/c/599587/8/specs/stein/approved/numa-aware-live-migration.rst@327
16:16:13 bauzas the question is : once we implement this, we will change how we will accept live migrations
16:16:28 bauzas should we signal it ?
16:16:52 sean-k-mooney bauzas: how do you mean
16:16:54 bauzas or just drop a release note saying "by Stein, NUMA-aware live migrations will be unaccepted"
16:17:01 artom bauzas, we could make it configurable, sort like what stephenfin is proposing with https://review.openstack.org/#/c/611088/
16:17:09 mriedem didn't this already come up in https://review.openstack.org/#/c/611088/
16:17:10 mriedem yeah
16:17:27 artom bauzas, but... why? Why would anyone want to live migrate in the middle of an upgrade with mixed computes?
16:17:42 artom Live migrate *knowing* that things will mostly likely go south
16:18:05 sean-k-mooney artom: it not that uncommon
16:18:17 bauzas artom: it's a possibility yes
16:18:23 bauzas that's even a common pattern
16:18:31 sean-k-mooney in that case they are likely specifyign the host as they are freeing up old host to upgrade
16:18:37 bauzas operators do a lot of migrations when they upgrade
16:18:48 dansmith artom: live migration during an upgrade is the primary mechanism that most people use
16:19:07 dansmith artom: *lots* of people refuse to upgrade a compute node until they've moved everything off of it
16:19:14 bauzas tbh, I'm fine with dropping a release note saying there will be an impact
16:19:21 artom dansmith, fair enough
16:19:49 bauzas given worloads should move from old to new
16:19:56 bauzas and not from new to old
16:19:59 sean-k-mooney bauzas: we could do what we did for multiple port bindings and just fall back to the old behavior
16:20:12 bauzas oh shit, the problem is with old to new
16:20:17 bauzas not the contrary, my bad
16:20:41 artom I just don't see how we can realistically handle that
16:20:44 bauzas so, yeah, we could shoot operators in the foot
16:20:58 bauzas artom: we could make a flag during the upgrade
16:21:03 artom Keep the old code paths intact while somehow adding claims and all the new XML stuff
16:21:13 bauzas and say 'if you need so, it's on you, folks"
16:21:21 sean-k-mooney artom: for old to new we can fallback to the old behvior
16:21:38 bauzas new to old isn't a problem
16:21:41 bauzas new to new isn't a problem
16:21:42 artom sean-k-mooney, can we? How do we even implement that?
16:21:45 bauzas old to new is the problem
16:22:00 sean-k-mooney e.g. claim nothing and tell people to migrate again after migrate to fix everything
16:22:02 bauzas so yeah, we somehow need to keep the broken behaviour
16:22:05 artom It's going to be way ugly with a whole bunch of conditionals all over the place
16:22:15 bauzas artom: I know, it's freaking ugly
16:22:46 artom Unless we do the new stuff in entirely new RPC calls/casts
16:22:46 bauzas but I don't see operators be happy with what you propose :p
16:22:55 openstack Launchpad bug 1805659 in OpenStack Compute (nova) "nova notifications hammering the message bus" [Undecided,New]
16:22:55 gibi mriedem, jmlowe: I reported https://bugs.launchpad.net/nova/+bug/1805659
16:22:58 sean-k-mooney artom: its what we did for multiple port bindings. we check if a field exitsing in the migrate data if not we do the old stuff if its there we do the new way
16:23:04 artom bauzas, yeah, probably true

Earlier   Later