Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-28
15:39:23 Sundar mriedem: dansmith: I am working on a POC, a simple one focusing only on the Nova - Cyborg calls. It needs official clearance within my company before I can share it
15:39:59 mriedem the point of the rest api was so that the data and drivers live within cyborg, not nova, and nova hits those drivers over the API
15:40:06 mriedem if we could put a shim in like PlacementDirect, great
15:40:07 dansmith Sundar: yeah that's not going to work very well for collab, but.. good luck :/
15:40:28 jangutter Sundar: I don't think that anybody disputes that it's better to have a separate entity to handle out-of-Nova scope things. The trick is that Nova already has a way to pass PCI devices to instances. There's also existing code in os-vif that gets called "on plug events".
15:40:32 mriedem we talked about all of this last week when we said we didn't need an os-acc library and could just use python-cyborgclient
15:40:34 sean-k-mooney dansmith: well i have some leaning towrad a libary solution too
15:40:46 mriedem and i was also confused why we needed to hit a rest api to get to the cyborg drivers, i.e. why it wasn't more like os-brick
15:41:13 dansmith I'm fine with the programming part being in a library instead of an agent if we want
15:41:37 dansmith mriedem: it's quite a bit more complicated than just manipulating standard system things to connect to an iscsi target
15:41:51 mriedem right that was explained last week
15:41:54 mriedem probably about this time :)
15:42:10 dansmith but I don't think the tracking of devices, workflow to program, clean, etc these devices belongs in nova
15:42:31 mriedem i don't really think we should build on that existing stuff within nova either,
15:42:47 mriedem b/c once something is built on it, it makes it harder to clean it up later (it already is taking forever)
15:42:53 mriedem and that pci code went in in juno
15:43:11 artom Just... pay a bunch of interns, and stick a CORBA thing in front.
15:43:28 mriedem i was going to add, "says the guy that isn't working on cleaning any of it up"
15:43:28 jangutter artom: don't joke, xml will come out.
15:43:28 jaypipes artom: lol
15:43:36 mriedem for myself
15:43:39 sean-k-mooney artom: just no...
15:44:01 mriedem if we're talking xml, CIM model model this all so nicely!
15:44:06 mriedem *would model
15:44:35 artom If no one can agree on what to do, maybe if we all agree on what *not* to do, and then do that...
15:46:10 sean-k-mooney i think at this point we would all like to see a POC rather then talk in abstract terms about it in a loop
15:46:22 dansmith yup
15:46:47 jangutter we're falling into the trap of agreeing on stuff: which means it won't happen.
15:46:49 jaypipes Sundar: what's the likelihood of your PoC being available this year?
15:46:58 jaypipes Sundar: I understand you need permission from legal?
15:48:28 Sundar I'd like some agreement on the general direction of the POC. At the PTG, there was alignment on defining REST calls, and I wrote that up in a spec that is being reviewed. The discussion here seems more philosophical. I suggest that the POC should focus on clarifying the spec.
15:49:28 artom Sundar, I think at some point you'll have to accept that work might get wasted :(
15:49:41 dansmith Sundar: while everything is just words in a document, the discussion can get pretty philosophical pretty quick, as you have no doubt noticed
15:49:44 artom I've been observing this discussion from afar since... Dublin?
15:49:48 sean-k-mooney Sundar: can we do a simple poc of booting a vm with just a fixed fuction pci device
15:49:54 Sundar jaypipes: Yes, it requires legal clearance. I may get a requirement to follow a process, or likely multiple processes. I am hoping that, confining it to upstream code and no real devices will make it go faster
15:50:12 artom Bite the bullet, write some code, share it
15:50:23 artom If the code gets scrapped and needs to be rewritte, so be it
15:50:24 sean-k-mooney Sundar: the intel fpga stuff may but nova and cyborg working together dont
15:50:26 jaypipes Sundar: that's generally not what PoCs are, but ok :) My suggestion is to demo Nova actually being able to /usr/bin/fpgaconf some device. If you want to go through the trouble of getting all the REST API stuff done in order for Nova to shell out to /usr/bin/fpgaconf, go for it. :)
15:53:36 jmlowe If anybody remembers my slow placement api problem from a couple of days ago, it seems that NAT in the 4.15 kernel that ships with ubuntu 18.04 is problematic at scale
15:54:41 cdent jmlowe: I am a) relieved to hear that, b) sorry for your luck
15:55:06 cdent can you define "at scale"?
15:56:03 mriedem cern runs centos right?
15:56:05 mriedem tssurya: ^
15:56:32 jmlowe I have a quick question about versioned notifications, if a notification has been transitioned to versioned and the setting is for legacy does that mean the notification will never be emitted?
15:57:10 mriedem jmlowe: what is the value of this option https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format
15:57:13 mriedem in your nova.conf
15:57:15 jmlowe cdent: I was falling over at ~280, threshold is probably significantly lower, ubuntu 16.04 didn't cause problems
15:57:20 gibi jmlowe: if you configured nova to emit legacy notifications only then nova only emts the legacy ones
15:58:01 gibi jmlowe: we did not removed the legacy notifications, and we did not plan to remove them
15:59:08 jmlowe I have unversioned, was looking at rocky release notes and saw that instance exits has been transitioned and that's the one event I really need to consume with ceilometer/panko which doesn't do versioned yet
15:59:18 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Support volume-backed server rebuild https://review.openstack.org/532407
15:59:41 mriedem jmlowe: just leave as unversioned then,
15:59:47 mriedem as gibi said, we haven't removed the unversioned notifications
16:00:04 gibi jmlowe: if https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format is configured to both or unversioned then nova still sends the old instance.exists
16:00:07 mriedem most, if not all, openstack projects that consume notifications from nova are still using unversoined notifications
16:01:07 mriedem dansmith: i'm +2 on the volume-backed rebuild spec now https://review.openstack.org/#/c/532407/
16:02:36 jmlowe ok, perfect, thanks, needed a sanity check to make sure before I happily upgraded myself into a sisyphean custom patch cycle
16:03:09 jmlowe mriedem: my exploding rabbit queues say otherwise
16:04:01 jmlowe mriedem: versioned notifications for ceilometer are still tagged as a wishlist bug
16:06:55 mriedem jmlowe: umm
16:07:06 mriedem if you're using unversioned, there should be no difference
16:07:23 Sundar jaypipes: I found the source for fpgaconf.c : https://github.com/OPAE/opae-sdk/tree/master/tools/base/fpgaconf
16:07:24 jmlowe mriedem: that was learned the hard way
16:07:40 mriedem jmlowe: can you expand? did you upgrade and the format option was using 'both'?
16:08:48 mriedem i remember godaddy saying in boston that the instance.exists notifications in particular hammer their MQ
16:09:59 sean-k-mooney Sundar: could we do a poc usign a FakeDriver the simulates a fake device and uses that to test this end to end and there for remove the hardware depency
16:10:26 sean-k-mooney we have asked this in the past
16:10:45 sean-k-mooney it woudl be useful for ci if nothing else
16:10:48 jmlowe Now on queens, upgraded in place starting from liberty, didn't really pay attention, just assumed everything would work, the versioned notifications queue grew without bound until I finally got smart an capped nova with legacy notifications
16:11:57 mriedem ok, so it was sending both then
16:12:07 jmlowe whatever the default is
16:12:11 mriedem it's 'both'
16:12:18 mriedem gibi: i almost wonder if we should change the default to unversioned...
16:12:41 mriedem i know we want people to use versioned, but there aren't any projects working on that
16:12:49 mriedem or,
16:13:05 mriedem maybe we should start a 'performance / scale considerations' doc in nova for stuff like this
16:13:28 mriedem "rabbit got you down? check your notifications settings."
16:14:35 gibi mriedem: does changing the default in a bugfix is safe from config compatibility perspective?
16:15:07 jmlowe That may thwart my ambitions to be a obscenely high paid consultant
16:15:19 bauzas mriedem: artom: there ? I have a concern on https://review.openstack.org/#/c/599587/
16:15:32 bauzas most of my comments are nits but the last one is important
16:15:46 bauzas artom: mriedem: https://review.openstack.org/#/c/599587/8/specs/stein/approved/numa-aware-live-migration.rst@327
16:16:13 bauzas the question is : once we implement this, we will change how we will accept live migrations
16:16:28 bauzas should we signal it ?
16:16:52 sean-k-mooney bauzas: how do you mean
16:16:54 bauzas or just drop a release note saying "by Stein, NUMA-aware live migrations will be unaccepted"
16:17:01 artom bauzas, we could make it configurable, sort like what stephenfin is proposing with https://review.openstack.org/#/c/611088/
16:17:09 mriedem didn't this already come up in https://review.openstack.org/#/c/611088/
16:17:10 mriedem yeah
16:17:27 artom bauzas, but... why? Why would anyone want to live migrate in the middle of an upgrade with mixed computes?
16:17:42 artom Live migrate *knowing* that things will mostly likely go south
16:18:05 sean-k-mooney artom: it not that uncommon
16:18:17 bauzas artom: it's a possibility yes
16:18:23 bauzas that's even a common pattern
16:18:31 sean-k-mooney in that case they are likely specifyign the host as they are freeing up old host to upgrade
16:18:37 bauzas operators do a lot of migrations when they upgrade
16:18:48 dansmith artom: live migration during an upgrade is the primary mechanism that most people use
16:19:07 dansmith artom: *lots* of people refuse to upgrade a compute node until they've moved everything off of it

Earlier   Later