| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-28 | |||
| 15:43:28 | jangutter | artom: don't joke, xml will come out. | |
| 15:43:28 | jaypipes | artom: lol | |
| 15:43:36 | mriedem | for myself | |
| 15:43:39 | sean-k-mooney | artom: just no... | |
| 15:44:01 | mriedem | if we're talking xml, CIM model model this all so nicely! | |
| 15:44:06 | mriedem | *would model | |
| 15:44:35 | artom | If no one can agree on what to do, maybe if we all agree on what *not* to do, and then do that... | |
| 15:46:10 | sean-k-mooney | i think at this point we would all like to see a POC rather then talk in abstract terms about it in a loop | |
| 15:46:22 | dansmith | yup | |
| 15:46:47 | jangutter | we're falling into the trap of agreeing on stuff: which means it won't happen. | |
| 15:46:49 | jaypipes | Sundar: what's the likelihood of your PoC being available this year? | |
| 15:46:58 | jaypipes | Sundar: I understand you need permission from legal? | |
| 15:48:28 | Sundar | I'd like some agreement on the general direction of the POC. At the PTG, there was alignment on defining REST calls, and I wrote that up in a spec that is being reviewed. The discussion here seems more philosophical. I suggest that the POC should focus on clarifying the spec. | |
| 15:49:28 | artom | Sundar, I think at some point you'll have to accept that work might get wasted :( | |
| 15:49:41 | dansmith | Sundar: while everything is just words in a document, the discussion can get pretty philosophical pretty quick, as you have no doubt noticed | |
| 15:49:44 | artom | I've been observing this discussion from afar since... Dublin? | |
| 15:49:48 | sean-k-mooney | Sundar: can we do a simple poc of booting a vm with just a fixed fuction pci device | |
| 15:49:54 | Sundar | jaypipes: Yes, it requires legal clearance. I may get a requirement to follow a process, or likely multiple processes. I am hoping that, confining it to upstream code and no real devices will make it go faster | |
| 15:50:12 | artom | Bite the bullet, write some code, share it | |
| 15:50:23 | artom | If the code gets scrapped and needs to be rewritte, so be it | |
| 15:50:24 | sean-k-mooney | Sundar: the intel fpga stuff may but nova and cyborg working together dont | |
| 15:50:26 | jaypipes | Sundar: that's generally not what PoCs are, but ok :) My suggestion is to demo Nova actually being able to /usr/bin/fpgaconf some device. If you want to go through the trouble of getting all the REST API stuff done in order for Nova to shell out to /usr/bin/fpgaconf, go for it. :) | |
| 15:53:36 | jmlowe | If anybody remembers my slow placement api problem from a couple of days ago, it seems that NAT in the 4.15 kernel that ships with ubuntu 18.04 is problematic at scale | |
| 15:54:41 | cdent | jmlowe: I am a) relieved to hear that, b) sorry for your luck | |
| 15:55:06 | cdent | can you define "at scale"? | |
| 15:56:03 | mriedem | cern runs centos right? | |
| 15:56:05 | mriedem | tssurya: ^ | |
| 15:56:32 | jmlowe | I have a quick question about versioned notifications, if a notification has been transitioned to versioned and the setting is for legacy does that mean the notification will never be emitted? | |
| 15:57:10 | mriedem | jmlowe: what is the value of this option https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format | |
| 15:57:13 | mriedem | in your nova.conf | |
| 15:57:15 | jmlowe | cdent: I was falling over at ~280, threshold is probably significantly lower, ubuntu 16.04 didn't cause problems | |
| 15:57:20 | gibi | jmlowe: if you configured nova to emit legacy notifications only then nova only emts the legacy ones | |
| 15:58:01 | gibi | jmlowe: we did not removed the legacy notifications, and we did not plan to remove them | |
| 15:59:08 | jmlowe | I have unversioned, was looking at rocky release notes and saw that instance exits has been transitioned and that's the one event I really need to consume with ceilometer/panko which doesn't do versioned yet | |
| 15:59:18 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Support volume-backed server rebuild https://review.openstack.org/532407 | |
| 15:59:41 | mriedem | jmlowe: just leave as unversioned then, | |
| 15:59:47 | mriedem | as gibi said, we haven't removed the unversioned notifications | |
| 16:00:04 | gibi | jmlowe: if https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format is configured to both or unversioned then nova still sends the old instance.exists | |
| 16:00:07 | mriedem | most, if not all, openstack projects that consume notifications from nova are still using unversoined notifications | |
| 16:01:07 | mriedem | dansmith: i'm +2 on the volume-backed rebuild spec now https://review.openstack.org/#/c/532407/ | |
| 16:02:36 | jmlowe | ok, perfect, thanks, needed a sanity check to make sure before I happily upgraded myself into a sisyphean custom patch cycle | |
| 16:03:09 | jmlowe | mriedem: my exploding rabbit queues say otherwise | |
| 16:04:01 | jmlowe | mriedem: versioned notifications for ceilometer are still tagged as a wishlist bug | |
| 16:06:55 | mriedem | jmlowe: umm | |
| 16:07:06 | mriedem | if you're using unversioned, there should be no difference | |
| 16:07:23 | Sundar | jaypipes: I found the source for fpgaconf.c : https://github.com/OPAE/opae-sdk/tree/master/tools/base/fpgaconf | |
| 16:07:24 | jmlowe | mriedem: that was learned the hard way | |
| 16:07:40 | mriedem | jmlowe: can you expand? did you upgrade and the format option was using 'both'? | |
| 16:08:48 | mriedem | i remember godaddy saying in boston that the instance.exists notifications in particular hammer their MQ | |
| 16:09:59 | sean-k-mooney | Sundar: could we do a poc usign a FakeDriver the simulates a fake device and uses that to test this end to end and there for remove the hardware depency | |
| 16:10:26 | sean-k-mooney | we have asked this in the past | |
| 16:10:45 | sean-k-mooney | it woudl be useful for ci if nothing else | |
| 16:10:48 | jmlowe | Now on queens, upgraded in place starting from liberty, didn't really pay attention, just assumed everything would work, the versioned notifications queue grew without bound until I finally got smart an capped nova with legacy notifications | |
| 16:11:57 | mriedem | ok, so it was sending both then | |
| 16:12:07 | jmlowe | whatever the default is | |
| 16:12:11 | mriedem | it's 'both' | |
| 16:12:18 | mriedem | gibi: i almost wonder if we should change the default to unversioned... | |
| 16:12:41 | mriedem | i know we want people to use versioned, but there aren't any projects working on that | |
| 16:12:49 | mriedem | or, | |
| 16:13:05 | mriedem | maybe we should start a 'performance / scale considerations' doc in nova for stuff like this | |
| 16:13:28 | mriedem | "rabbit got you down? check your notifications settings." | |
| 16:14:35 | gibi | mriedem: does changing the default in a bugfix is safe from config compatibility perspective? | |
| 16:15:07 | jmlowe | That may thwart my ambitions to be a obscenely high paid consultant | |
| 16:15:19 | bauzas | mriedem: artom: there ? I have a concern on https://review.openstack.org/#/c/599587/ | |
| 16:15:32 | bauzas | most of my comments are nits but the last one is important | |
| 16:15:46 | bauzas | artom: mriedem: https://review.openstack.org/#/c/599587/8/specs/stein/approved/numa-aware-live-migration.rst@327 | |
| 16:16:13 | bauzas | the question is : once we implement this, we will change how we will accept live migrations | |
| 16:16:28 | bauzas | should we signal it ? | |
| 16:16:52 | sean-k-mooney | bauzas: how do you mean | |
| 16:16:54 | bauzas | or just drop a release note saying "by Stein, NUMA-aware live migrations will be unaccepted" | |
| 16:17:01 | artom | bauzas, we could make it configurable, sort like what stephenfin is proposing with https://review.openstack.org/#/c/611088/ | |
| 16:17:09 | mriedem | didn't this already come up in https://review.openstack.org/#/c/611088/ | |
| 16:17:10 | mriedem | yeah | |
| 16:17:27 | artom | bauzas, but... why? Why would anyone want to live migrate in the middle of an upgrade with mixed computes? | |
| 16:17:42 | artom | Live migrate *knowing* that things will mostly likely go south | |
| 16:18:05 | sean-k-mooney | artom: it not that uncommon | |
| 16:18:17 | bauzas | artom: it's a possibility yes | |
| 16:18:23 | bauzas | that's even a common pattern | |
| 16:18:31 | sean-k-mooney | in that case they are likely specifyign the host as they are freeing up old host to upgrade | |
| 16:18:37 | bauzas | operators do a lot of migrations when they upgrade | |
| 16:18:48 | dansmith | artom: live migration during an upgrade is the primary mechanism that most people use | |
| 16:19:07 | dansmith | artom: *lots* of people refuse to upgrade a compute node until they've moved everything off of it | |
| 16:19:14 | bauzas | tbh, I'm fine with dropping a release note saying there will be an impact | |
| 16:19:21 | artom | dansmith, fair enough | |
| 16:19:49 | bauzas | given worloads should move from old to new | |
| 16:19:56 | bauzas | and not from new to old | |
| 16:19:59 | sean-k-mooney | bauzas: we could do what we did for multiple port bindings and just fall back to the old behavior | |
| 16:20:12 | bauzas | oh shit, the problem is with old to new | |
| 16:20:17 | bauzas | not the contrary, my bad | |
| 16:20:41 | artom | I just don't see how we can realistically handle that | |
| 16:20:44 | bauzas | so, yeah, we could shoot operators in the foot | |
| 16:20:58 | bauzas | artom: we could make a flag during the upgrade | |
| 16:21:03 | artom | Keep the old code paths intact while somehow adding claims and all the new XML stuff | |
| 16:21:13 | bauzas | and say 'if you need so, it's on you, folks" | |
| 16:21:21 | sean-k-mooney | artom: for old to new we can fallback to the old behvior | |
| 16:21:38 | bauzas | new to old isn't a problem | |
| 16:21:41 | bauzas | new to new isn't a problem | |
| 16:21:42 | artom | sean-k-mooney, can we? How do we even implement that? | |
| 16:21:45 | bauzas | old to new is the problem | |
| 16:22:00 | sean-k-mooney | e.g. claim nothing and tell people to migrate again after migrate to fix everything | |