| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-11 | |||
| 14:38:42 | mriedem | jkulik: as far as i know vmware hasn't had an "upstream" team in awhile | |
| 14:38:49 | mriedem | cdent was the last closest person to that for nova at least | |
| 14:39:01 | dansmith | and honestly, it has always been a fight to keep them involved, which is pretty exhausting | |
| 14:39:33 | mriedem | i'm sure that team would say it was exhausting dealing with us as well :) | |
| 14:39:45 | mriedem | "what do you mean my bug fix needs tests?!" | |
| 14:41:39 | jkulik | as far as I heard: yes, they found it exhausting. | |
| 14:41:48 | mriedem | heh | |
| 14:41:57 | dansmith | yeah I have no doubt | |
| 14:42:03 | mriedem | i mean, quality control, who needs it | |
| 14:42:04 | mriedem | psh | |
| 14:42:17 | jkulik | main problem seems to be, that they have their own product based on openstack which works fine for them | |
| 14:42:36 | mriedem | VIO | |
| 14:42:39 | mriedem | right? | |
| 14:42:40 | jkulik | yes | |
| 14:42:45 | mriedem | sure, with patches | |
| 14:42:46 | dansmith | and that's cool, but we don't need to keep a broken driver in our tree, especially with people asking on the ML why it doesn't work | |
| 14:42:53 | mriedem | so you buy that thing and you're stuck with them for support b/c of the patches | |
| 14:43:06 | jkulik | sure, they need to make money off it | |
| 14:43:12 | jkulik | otherwise everybody could just install openstack ;) | |
| 14:43:40 | sean-k-mooney | jkulik: our they could enforce an upstream first poicy for all freatres and backports | |
| 14:43:47 | sean-k-mooney | *or | |
| 14:43:57 | jkulik | which they obviously still make money from, because you'd still have VMware hypervisors and stuff | |
| 14:44:17 | jkulik | tbh, upstream first is hard. | |
| 14:44:50 | sean-k-mooney | it can be but its generally worth it. i thnik the quality of the final solution is typically better | |
| 14:45:24 | sean-k-mooney | but it is a higher barrier to entry for enableing a feature | |
| 14:45:26 | jkulik | I agree. But some things you can't upstream and once you're down that road, you're not that willing to upstream the rest. | |
| 14:45:29 | mriedem | depends on where you want to invest time and money, up front or on the backside dealing with maintaining a fork | |
| 14:46:28 | dansmith | jkulik: we're not arguing that they shouldn't have a downstream.. we're arguing that if they want their base in the upstream, there's a minimum bar and we're not going to do their maintenance for free, that's all | |
| 14:46:29 | dansmith | we all make money from openstack one way or the other, nobody here doesn't recognize that | |
| 14:46:29 | mriedem | and getting the team culture in place for people that aren't used to having their code reviewed | |
| 14:46:49 | mriedem | i make $0 from openstack now | |
| 14:46:58 | mriedem | wtf am i even doing in this conversation? :) | |
| 14:47:07 | jkulik | sorry :D | |
| 14:47:17 | dansmith | mriedem: well, the 5th year senior that still hangs around campus being the exception :) | |
| 14:47:41 | mriedem | i get older, nova stays the same....wait | |
| 14:47:46 | lyarwood | haha | |
| 14:47:47 | dansmith | LOLOL | |
| 14:47:54 | jkulik | I think my main problem is, that I can't convince VMware to do the maintenance upstream and need that driver to work ;) | |
| 14:48:15 | mriedem | jkulik: so can SAP have one or two developers that start working on it? | |
| 14:48:28 | mriedem | it's not like vmware is the only company that can work on that driver | |
| 14:48:30 | dansmith | jkulik: I want lots of things for free too | |
| 14:48:33 | mriedem | it's been on life support for years | |
| 14:49:25 | sean-k-mooney | jkulik: do you consume the driver directly form upstream or via the vio product | |
| 14:49:26 | jkulik | I can talk to my managers about it, but given our team size, it'll basically still be life-support. | |
| 14:49:34 | jkulik | from upstream directly | |
| 14:49:54 | sean-k-mooney | ok makes sense | |
| 14:51:07 | dansmith | jkulik: you know every time we make a change to the virt drivers, we have to make a change to the vmware one, and with no tests, we don't know if it works or not right? that's a lot of burden for us, and if you look at all the changes to that driver in the last two years, it's just that.. guesses. | |
| 14:51:31 | jkulik | dansmith: yes. I totally get that. | |
| 14:51:50 | jkulik | in addition, VMware doesn't work like libvirt at all in too many cases. | |
| 14:52:02 | dansmith | yup :/ | |
| 14:53:25 | sean-k-mooney | the current vmware driver talks to vspher too right rather then to esxi | |
| 14:53:34 | mriedem | at least we don't have the 1:M host:node thing with that driver anymore | |
| 14:53:44 | mriedem | doesn't mean a single node isn't hosting 1K instances which crash the RT | |
| 14:54:01 | sean-k-mooney | so while esxi can be used without a licnce vsphere cant so we cant do first party ci even if we and the capastity to maintain it | |
| 14:54:13 | jkulik | the driver talks to vsphere, one nova node is a cluster (multiple hypervisors), but nova doesn't know about it | |
| 14:54:13 | mriedem | which has been a major pain point with the vmware team over the years, trying to change the nova architecture to handle that type of scale on a single node | |
| 14:54:19 | mriedem | jkulik: yeah i know | |
| 14:54:31 | mriedem | to nova it's a node with several thousand VMs potentially | |
| 14:54:39 | jkulik | yes | |
| 14:54:41 | mriedem | unlike libvirt with maybe a couple dozen | |
| 14:54:44 | dansmith | ...which is why I'm dubious about all the placement stuff really working these days | |
| 14:55:07 | jkulik | we recently ran into a problem with that concept, trying to deploy really big VMs | |
| 14:55:17 | sean-k-mooney | dansmith: well we added the same host migrate thing last cycle right? | |
| 14:55:22 | hemna | so the driver won't work in U ? | |
| 14:55:25 | sean-k-mooney | so at least at that point it worked | |
| 14:55:38 | sean-k-mooney | but ya not sure placment will imporve performance or schduling in anyway | |
| 14:55:46 | dansmith | hemna: we have no idea | |
| 14:55:55 | dansmith | sean-k-mooney: which? | |
| 14:56:19 | jkulik | which is the main pain point ... not knowing if stuff even works | |
| 14:56:33 | hemna | sorry I haven't been following nova as much lately, but has there been a change that would cause the driver to not work? | |
| 14:56:45 | dansmith | hemna: lots of them | |
| 14:57:12 | hemna | :( | |
| 14:58:34 | dansmith | sean-k-mooney: actually the second to last one had a report from vmware ci that said it failed | |
| 14:58:46 | dansmith | sean-k-mooney: https://review.opendev.org/#/c/681004/ | |
| 14:59:01 | mriedem | hemna: you know how cinder has/had a pretty aggressive rule about deprecating volume drivers with no reporting/reliable ci in just a single release? | |
| 14:59:20 | mriedem | the vmware ci has been pretty much a no show for *years* | |
| 14:59:20 | hemna | yah, I think we are changing from that though slowly | |
| 14:59:24 | hemna | :( | |
| 14:59:46 | mriedem | sure, because fewer maintainers to care | |
| 14:59:52 | mriedem | and cinder also has like 100 damn drivers | |
| 14:59:53 | hemna | ok, well it would be in our interest to keep it in tree, so maybe we can twist some arms with our support with vmware | |
| 14:59:56 | sean-k-mooney | dansmith: i was refering to mriedem COMPUTE_SAME_HOST_COLD_MIGRATE changes https://github.com/openstack/nova/commit/4921e822e73383af0c8da4c5e3acfaa021eafe68 | |
| 15:00:16 | sean-k-mooney | dansmith: that was the last entirly vmware specific feature i recall | |
| 15:00:17 | dansmith | sean-k-mooney: vmware failed on that | |
| 15:00:28 | sean-k-mooney | oh ok | |
| 15:00:46 | dansmith | I don't think that was for vmware specifically, IIRC | |
| 15:00:51 | mriedem | it wasn't | |
| 15:01:04 | mriedem | it's just the only driver that will report that trait | |
| 15:01:10 | dansmith | right | |
| 15:01:38 | mriedem | for libvirt it means you can configure the api to allow same-host RESIZE but prevent the API from telling the scheduler "oh sure send cold migrations there also" | |
| 15:01:39 | sean-k-mooney | ah ok i was just looking at the release note and it mentioned ``VMwareVCDriver`` sepecifclaly | |
| 15:01:40 | mriedem | and then those fail | |
| 15:02:10 | sean-k-mooney | ah ok | |
| 15:02:28 | dansmith | sean-k-mooney: I think this was because we knew we had broken their same-host-migrate requirement so we were trying to add a thing to make it possible for them again | |
| 15:03:12 | sean-k-mooney | right which is an example of use trying to maintain a driver we cant properly test | |
| 15:03:27 | sean-k-mooney | /use/us | |
| 15:03:41 | dansmith | yup, | |
| 15:04:10 | dansmith | so we definitely broke them, and then merged something to try to unbreak them, but without tests, hard to say if it even works | |
| 15:04:12 | jkulik | what I get from the discussion: our team should interact more with upstream and we should try to create a CI ASAP ourselves, if we want the driver to continue to live upstream | |
| 15:04:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_MULTIATTACH https://review.opendev.org/710238 | |