| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-11 | |||
| 14:49:34 | jkulik | from upstream directly | |
| 14:49:54 | sean-k-mooney | ok makes sense | |
| 14:51:07 | dansmith | jkulik: you know every time we make a change to the virt drivers, we have to make a change to the vmware one, and with no tests, we don't know if it works or not right? that's a lot of burden for us, and if you look at all the changes to that driver in the last two years, it's just that.. guesses. | |
| 14:51:31 | jkulik | dansmith: yes. I totally get that. | |
| 14:51:50 | jkulik | in addition, VMware doesn't work like libvirt at all in too many cases. | |
| 14:52:02 | dansmith | yup :/ | |
| 14:53:25 | sean-k-mooney | the current vmware driver talks to vspher too right rather then to esxi | |
| 14:53:34 | mriedem | at least we don't have the 1:M host:node thing with that driver anymore | |
| 14:53:44 | mriedem | doesn't mean a single node isn't hosting 1K instances which crash the RT | |
| 14:54:01 | sean-k-mooney | so while esxi can be used without a licnce vsphere cant so we cant do first party ci even if we and the capastity to maintain it | |
| 14:54:13 | jkulik | the driver talks to vsphere, one nova node is a cluster (multiple hypervisors), but nova doesn't know about it | |
| 14:54:13 | mriedem | which has been a major pain point with the vmware team over the years, trying to change the nova architecture to handle that type of scale on a single node | |
| 14:54:19 | mriedem | jkulik: yeah i know | |
| 14:54:31 | mriedem | to nova it's a node with several thousand VMs potentially | |
| 14:54:39 | jkulik | yes | |
| 14:54:41 | mriedem | unlike libvirt with maybe a couple dozen | |
| 14:54:44 | dansmith | ...which is why I'm dubious about all the placement stuff really working these days | |
| 14:55:07 | jkulik | we recently ran into a problem with that concept, trying to deploy really big VMs | |
| 14:55:17 | sean-k-mooney | dansmith: well we added the same host migrate thing last cycle right? | |
| 14:55:22 | hemna | so the driver won't work in U ? | |
| 14:55:25 | sean-k-mooney | so at least at that point it worked | |
| 14:55:38 | sean-k-mooney | but ya not sure placment will imporve performance or schduling in anyway | |
| 14:55:46 | dansmith | hemna: we have no idea | |
| 14:55:55 | dansmith | sean-k-mooney: which? | |
| 14:56:19 | jkulik | which is the main pain point ... not knowing if stuff even works | |
| 14:56:33 | hemna | sorry I haven't been following nova as much lately, but has there been a change that would cause the driver to not work? | |
| 14:56:45 | dansmith | hemna: lots of them | |
| 14:57:12 | hemna | :( | |
| 14:58:34 | dansmith | sean-k-mooney: actually the second to last one had a report from vmware ci that said it failed | |
| 14:58:46 | dansmith | sean-k-mooney: https://review.opendev.org/#/c/681004/ | |
| 14:59:01 | mriedem | hemna: you know how cinder has/had a pretty aggressive rule about deprecating volume drivers with no reporting/reliable ci in just a single release? | |
| 14:59:20 | mriedem | the vmware ci has been pretty much a no show for *years* | |
| 14:59:20 | hemna | yah, I think we are changing from that though slowly | |
| 14:59:24 | hemna | :( | |
| 14:59:46 | mriedem | sure, because fewer maintainers to care | |
| 14:59:52 | mriedem | and cinder also has like 100 damn drivers | |
| 14:59:53 | hemna | ok, well it would be in our interest to keep it in tree, so maybe we can twist some arms with our support with vmware | |
| 14:59:56 | sean-k-mooney | dansmith: i was refering to mriedem COMPUTE_SAME_HOST_COLD_MIGRATE changes https://github.com/openstack/nova/commit/4921e822e73383af0c8da4c5e3acfaa021eafe68 | |
| 15:00:16 | sean-k-mooney | dansmith: that was the last entirly vmware specific feature i recall | |
| 15:00:17 | dansmith | sean-k-mooney: vmware failed on that | |
| 15:00:28 | sean-k-mooney | oh ok | |
| 15:00:46 | dansmith | I don't think that was for vmware specifically, IIRC | |
| 15:00:51 | mriedem | it wasn't | |
| 15:01:04 | mriedem | it's just the only driver that will report that trait | |
| 15:01:10 | dansmith | right | |
| 15:01:38 | mriedem | for libvirt it means you can configure the api to allow same-host RESIZE but prevent the API from telling the scheduler "oh sure send cold migrations there also" | |
| 15:01:39 | sean-k-mooney | ah ok i was just looking at the release note and it mentioned ``VMwareVCDriver`` sepecifclaly | |
| 15:01:40 | mriedem | and then those fail | |
| 15:02:10 | sean-k-mooney | ah ok | |
| 15:02:28 | dansmith | sean-k-mooney: I think this was because we knew we had broken their same-host-migrate requirement so we were trying to add a thing to make it possible for them again | |
| 15:03:12 | sean-k-mooney | right which is an example of use trying to maintain a driver we cant properly test | |
| 15:03:27 | sean-k-mooney | /use/us | |
| 15:03:41 | dansmith | yup, | |
| 15:04:10 | dansmith | so we definitely broke them, and then merged something to try to unbreak them, but without tests, hard to say if it even works | |
| 15:04:12 | jkulik | what I get from the discussion: our team should interact more with upstream and we should try to create a CI ASAP ourselves, if we want the driver to continue to live upstream | |
| 15:04:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_MULTIATTACH https://review.opendev.org/710238 | |
| 15:05:03 | jkulik | and keep it running :) | |
| 15:05:14 | mriedem | tbc https://review.opendev.org/#/c/695220/ wasn't about a placement related regression, | |
| 15:05:20 | mriedem | it was a latent bug since forever | |
| 15:05:32 | mriedem | i just finally got around to fixing it *using* placement | |
| 15:06:00 | sean-k-mooney | sure you were fixing https://bugs.launchpad.net/nova/+bug/1748697 | |
| 15:06:00 | openstack | Launchpad bug 1748697 in OpenStack Compute (nova) "Cold migration fails when the filter only returns the host where the vm is located and the vm status is set to ERROR" [Low,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 15:06:16 | dansmith | yup | |
| 15:06:35 | stephenfin | lyarwood: got the last of those os-hypervisors backports for you here, if you have time https://review.opendev.org/#/c/711670/ | |
| 15:06:55 | gibi | jkulik: if you can show some movement around that CI during early Victoria then I think that will help us not deleting the driver from tree | |
| 15:07:15 | lyarwood | stephenfin: ack'd | |
| 15:11:21 | sean-k-mooney | it looks like there is a limvied free verion of vsphere too https://www.vmware.com/products/vsphere-hypervisor.html im not sure if that would be useable for ci but if the end user lisence agreement allowed it it might be possible to create a first party ci. | |
| 15:11:30 | hemna | dansmith do you guys have a page/docs for setting up a nova CI? in case we have to try and do it ourselves to report upstream | |
| 15:12:13 | dansmith | hemna: I think the infra team might have some general docs, but I'm not sure we have nova specific ones.. but I'm really the wrong person to ask | |
| 15:12:16 | gibi | hemna: there is a generic infra doc https://docs.openstack.org/infra/system-config/third_party.html | |
| 15:12:45 | hemna | ok thanks. I had setup a cinder CI ages ago and it's non trivial to manage | |
| 15:12:46 | sean-k-mooney | the zull docs are pretty useful too. | |
| 15:27:21 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add online migration for legacy NUMA objects https://review.opendev.org/537414 | |
| 15:27:32 | mriedem | hemna: there are also some relatively recent blogs that i've seen, e.g. https://01.org/openstack/blogs/manjeets/2019/how-test-open-source-hardware-drivers-zuul-v3-and-docker-compose | |
| 15:27:43 | mriedem | from what i remember it sounded like zuulv3 made it quite a bit simpler | |
| 15:28:29 | hemna | mriedem thank you. I'll check it out | |
| 15:28:30 | sean-k-mooney | mriedem: they have an example docker compose that deploy zuul/nodepool/gerrit and all the other infra for dev | |
| 15:28:58 | hemna | sean-k-mooney that would be nice | |
| 15:29:18 | sean-k-mooney | which is useful if you want to work on createign a job but i havent read the blog so im not sure if they have a production guide | |
| 15:31:14 | gibi | dansmith: could you add me to the NovaDrivers launchpad group | |
| 15:31:31 | dansmith | gibi: I dunno, can I? :) | |
| 15:31:51 | gibi | at least you are in the group | |
| 15:32:32 | gibi | efried_gone, mriedem: ^^ could you help if dansmith cannot? | |
| 15:32:44 | dansmith | gibi: "owner: openstack administrators" | |
| 15:32:47 | dansmith | I don't think we can do it | |
| 15:32:55 | dansmith | what do you need it for? | |
| 15:33:25 | gibi | according to the process we need LP milestones defined (at least we had for train) but we don't have them for ussuri at all | |
| 15:33:36 | dansmith | ah okay | |
| 15:33:54 | gibi | btw https://launchpad.net/~nova-drivers/+members shows that you are a member but not an admin. so I guess I need and admin | |
| 15:35:23 | dansmith | Yep, I can't add | |
| 15:35:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add online migration for legacy NUMA objects https://review.opendev.org/537414 | |
| 15:35:34 | gibi | dansmith: thanks for trying | |
| 15:36:04 | gibi | melwitt: could you add me to https://launchpad.net/~nova-drivers/+members ? | |
| 15:36:13 | mriedem | i can sec | |
| 15:36:33 | gibi | mriedem: thanks! | |
| 15:36:34 | mriedem | crap, "Claudio Hollanda (gibi) has been added as a member of this team." | |
| 15:36:38 | dansmith | seems like it's only these people https://launchpad.net/~openstack-admins | |
| 15:36:47 | mriedem | double crap "Dobos Balázs (balazs) has been added as a member of this team." | |
| 15:36:47 | mriedem | double crap "Dobos Balázs (balazs) has been added as a member of this team." | |
| 15:37:00 | mriedem | gibi: what is your launchpad id? | |