| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-11 | |||
| 14:37:06 | dansmith | working in queens I totally believe | |
| 14:37:58 | jkulik | fyi, we (SAP) reached out to VMware to get a statement on their upstream approach. they weren't ready for one, yet. | |
| 14:38:05 | mriedem | i forget her name but i was sending emails about the lack of CI back when the old project manager was involved | |
| 14:38:10 | dansmith | IMHO, to keep it we need more than just CI too.. we need support for devstack problems, like the one referenced on the mailing list and general confidence that it's working for more than just trivial CI configurations | |
| 14:38:42 | mriedem | jkulik: as far as i know vmware hasn't had an "upstream" team in awhile | |
| 14:38:49 | mriedem | cdent was the last closest person to that for nova at least | |
| 14:39:01 | dansmith | and honestly, it has always been a fight to keep them involved, which is pretty exhausting | |
| 14:39:33 | mriedem | i'm sure that team would say it was exhausting dealing with us as well :) | |
| 14:39:45 | mriedem | "what do you mean my bug fix needs tests?!" | |
| 14:41:39 | jkulik | as far as I heard: yes, they found it exhausting. | |
| 14:41:48 | mriedem | heh | |
| 14:41:57 | dansmith | yeah I have no doubt | |
| 14:42:03 | mriedem | i mean, quality control, who needs it | |
| 14:42:04 | mriedem | psh | |
| 14:42:17 | jkulik | main problem seems to be, that they have their own product based on openstack which works fine for them | |
| 14:42:36 | mriedem | VIO | |
| 14:42:39 | mriedem | right? | |
| 14:42:40 | jkulik | yes | |
| 14:42:45 | mriedem | sure, with patches | |
| 14:42:46 | dansmith | and that's cool, but we don't need to keep a broken driver in our tree, especially with people asking on the ML why it doesn't work | |
| 14:42:53 | mriedem | so you buy that thing and you're stuck with them for support b/c of the patches | |
| 14:43:06 | jkulik | sure, they need to make money off it | |
| 14:43:12 | jkulik | otherwise everybody could just install openstack ;) | |
| 14:43:40 | sean-k-mooney | jkulik: our they could enforce an upstream first poicy for all freatres and backports | |
| 14:43:47 | sean-k-mooney | *or | |
| 14:43:57 | jkulik | which they obviously still make money from, because you'd still have VMware hypervisors and stuff | |
| 14:44:17 | jkulik | tbh, upstream first is hard. | |
| 14:44:50 | sean-k-mooney | it can be but its generally worth it. i thnik the quality of the final solution is typically better | |
| 14:45:24 | sean-k-mooney | but it is a higher barrier to entry for enableing a feature | |
| 14:45:26 | jkulik | I agree. But some things you can't upstream and once you're down that road, you're not that willing to upstream the rest. | |
| 14:45:29 | mriedem | depends on where you want to invest time and money, up front or on the backside dealing with maintaining a fork | |
| 14:46:28 | dansmith | jkulik: we're not arguing that they shouldn't have a downstream.. we're arguing that if they want their base in the upstream, there's a minimum bar and we're not going to do their maintenance for free, that's all | |
| 14:46:29 | dansmith | we all make money from openstack one way or the other, nobody here doesn't recognize that | |
| 14:46:29 | mriedem | and getting the team culture in place for people that aren't used to having their code reviewed | |
| 14:46:49 | mriedem | i make $0 from openstack now | |
| 14:46:58 | mriedem | wtf am i even doing in this conversation? :) | |
| 14:47:07 | jkulik | sorry :D | |
| 14:47:17 | dansmith | mriedem: well, the 5th year senior that still hangs around campus being the exception :) | |
| 14:47:41 | mriedem | i get older, nova stays the same....wait | |
| 14:47:46 | lyarwood | haha | |
| 14:47:47 | dansmith | LOLOL | |
| 14:47:54 | jkulik | I think my main problem is, that I can't convince VMware to do the maintenance upstream and need that driver to work ;) | |
| 14:48:15 | mriedem | jkulik: so can SAP have one or two developers that start working on it? | |
| 14:48:28 | mriedem | it's not like vmware is the only company that can work on that driver | |
| 14:48:30 | dansmith | jkulik: I want lots of things for free too | |
| 14:48:33 | mriedem | it's been on life support for years | |
| 14:49:25 | sean-k-mooney | jkulik: do you consume the driver directly form upstream or via the vio product | |
| 14:49:26 | jkulik | I can talk to my managers about it, but given our team size, it'll basically still be life-support. | |
| 14:49:34 | jkulik | from upstream directly | |
| 14:49:54 | sean-k-mooney | ok makes sense | |
| 14:51:07 | dansmith | jkulik: you know every time we make a change to the virt drivers, we have to make a change to the vmware one, and with no tests, we don't know if it works or not right? that's a lot of burden for us, and if you look at all the changes to that driver in the last two years, it's just that.. guesses. | |
| 14:51:31 | jkulik | dansmith: yes. I totally get that. | |
| 14:51:50 | jkulik | in addition, VMware doesn't work like libvirt at all in too many cases. | |
| 14:52:02 | dansmith | yup :/ | |
| 14:53:25 | sean-k-mooney | the current vmware driver talks to vspher too right rather then to esxi | |
| 14:53:34 | mriedem | at least we don't have the 1:M host:node thing with that driver anymore | |
| 14:53:44 | mriedem | doesn't mean a single node isn't hosting 1K instances which crash the RT | |
| 14:54:01 | sean-k-mooney | so while esxi can be used without a licnce vsphere cant so we cant do first party ci even if we and the capastity to maintain it | |
| 14:54:13 | jkulik | the driver talks to vsphere, one nova node is a cluster (multiple hypervisors), but nova doesn't know about it | |
| 14:54:13 | mriedem | which has been a major pain point with the vmware team over the years, trying to change the nova architecture to handle that type of scale on a single node | |
| 14:54:19 | mriedem | jkulik: yeah i know | |
| 14:54:31 | mriedem | to nova it's a node with several thousand VMs potentially | |
| 14:54:39 | jkulik | yes | |
| 14:54:41 | mriedem | unlike libvirt with maybe a couple dozen | |
| 14:54:44 | dansmith | ...which is why I'm dubious about all the placement stuff really working these days | |
| 14:55:07 | jkulik | we recently ran into a problem with that concept, trying to deploy really big VMs | |
| 14:55:17 | sean-k-mooney | dansmith: well we added the same host migrate thing last cycle right? | |
| 14:55:22 | hemna | so the driver won't work in U ? | |
| 14:55:25 | sean-k-mooney | so at least at that point it worked | |
| 14:55:38 | sean-k-mooney | but ya not sure placment will imporve performance or schduling in anyway | |
| 14:55:46 | dansmith | hemna: we have no idea | |
| 14:55:55 | dansmith | sean-k-mooney: which? | |
| 14:56:19 | jkulik | which is the main pain point ... not knowing if stuff even works | |
| 14:56:33 | hemna | sorry I haven't been following nova as much lately, but has there been a change that would cause the driver to not work? | |
| 14:56:45 | dansmith | hemna: lots of them | |
| 14:57:12 | hemna | :( | |
| 14:58:34 | dansmith | sean-k-mooney: actually the second to last one had a report from vmware ci that said it failed | |
| 14:58:46 | dansmith | sean-k-mooney: https://review.opendev.org/#/c/681004/ | |
| 14:59:01 | mriedem | hemna: you know how cinder has/had a pretty aggressive rule about deprecating volume drivers with no reporting/reliable ci in just a single release? | |
| 14:59:20 | mriedem | the vmware ci has been pretty much a no show for *years* | |
| 14:59:20 | hemna | yah, I think we are changing from that though slowly | |
| 14:59:24 | hemna | :( | |
| 14:59:46 | mriedem | sure, because fewer maintainers to care | |
| 14:59:52 | mriedem | and cinder also has like 100 damn drivers | |
| 14:59:53 | hemna | ok, well it would be in our interest to keep it in tree, so maybe we can twist some arms with our support with vmware | |
| 14:59:56 | sean-k-mooney | dansmith: i was refering to mriedem COMPUTE_SAME_HOST_COLD_MIGRATE changes https://github.com/openstack/nova/commit/4921e822e73383af0c8da4c5e3acfaa021eafe68 | |
| 15:00:16 | sean-k-mooney | dansmith: that was the last entirly vmware specific feature i recall | |
| 15:00:17 | dansmith | sean-k-mooney: vmware failed on that | |
| 15:00:28 | sean-k-mooney | oh ok | |
| 15:00:46 | dansmith | I don't think that was for vmware specifically, IIRC | |
| 15:00:51 | mriedem | it wasn't | |
| 15:01:04 | mriedem | it's just the only driver that will report that trait | |
| 15:01:10 | dansmith | right | |
| 15:01:38 | mriedem | for libvirt it means you can configure the api to allow same-host RESIZE but prevent the API from telling the scheduler "oh sure send cold migrations there also" | |
| 15:01:39 | sean-k-mooney | ah ok i was just looking at the release note and it mentioned ``VMwareVCDriver`` sepecifclaly | |
| 15:01:40 | mriedem | and then those fail | |
| 15:02:10 | sean-k-mooney | ah ok | |
| 15:02:28 | dansmith | sean-k-mooney: I think this was because we knew we had broken their same-host-migrate requirement so we were trying to add a thing to make it possible for them again | |
| 15:03:12 | sean-k-mooney | right which is an example of use trying to maintain a driver we cant properly test | |
| 15:03:27 | sean-k-mooney | /use/us | |