| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-21 | |||
| 16:39:33 | bauzas | the problem here is that most of the times, someone pops up when we are about to delete a virt driver, asks us to not remove it, tells us we will have a 3rd-party CI for a long time, but then after a few months, disappears while the CI job has problems | |
| 16:39:56 | bauzas | this is really a matter of trusting people and companies | |
| 16:40:56 | bauzas | in the maintime, some users think they can use that virt driver and find some problems | |
| 16:40:57 | gibi | fwiesel: what woudl be your timeline setting up a the CI? | |
| 16:41:10 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: Remove nova-emulation from check pipeline of Yoga https://review.opendev.org/c/openstack/nova/+/901604 | |
| 16:41:19 | bauzas | this is why we now say those kind of drivers are experimental | |
| 16:42:09 | grandchild | hi, also from SAP here, and we understand the lack of trust. not much we can say to that now, we'll have to show our work first. | |
| 16:42:36 | grandchild | the issue is exactly the time frame for us to set up a CI while you (understandably) go ahead with removal. | |
| 16:42:44 | fwiesel | Sure, trust must be earned, so we can't expect you do that right away. And I can't expect the driver not to be removed considering current timelines. But hopefully we can find a way forward to re-integrate it then later under some better circumstances. | |
| 16:43:06 | bauzas | fwiesel: grandchild: thanks for replying | |
| 16:43:11 | bauzas | so, I have a question | |
| 16:43:28 | bauzas | say you need some time for running a 3rd-party CI | |
| 16:43:36 | bauzas | and you need to reply to gibi on that | |
| 16:43:41 | bauzas | then, | |
| 16:44:06 | bauzas | how would you be able to provide solutions for the tests that would have problems ? | |
| 16:44:15 | bauzas | do you know about the vmwareapi driver? | |
| 16:44:41 | bauzas | running a job is possible, making it fiable is more difficult | |
| 16:44:44 | grandchild | if you ask about our familiarity with the vmware driver code, i'd say it's fairly hih | |
| 16:44:46 | fwiesel | gibi: It is hard for us yet to give a timeline on the CI as we are yet unclear about how much effort it will be for us. | |
| 16:44:56 | bauzas | s/fiable/reliable | |
| 16:45:03 | gibi | fwiesel: I see | |
| 16:45:51 | fwiesel | We maintain a fork here: https://github.com/sapcc/nova | |
| 16:46:30 | fwiesel | Mostly because we try to work around issues of scalability in vmware which are probably not of interest to the general community. | |
| 16:46:32 | bauzas | fwiesel: maybe if you could tell us how long you would need to find the needed efforts, this would be nice | |
| 16:46:47 | bauzas | have you already run tempest against your fork ? | |
| 16:47:38 | fwiesel | bauzas: Yes, we run tempest and unit tests. We have disable some tempest tests, I can check which ones. | |
| 16:48:35 | sean-k-mooney | fwiesel: what do run them with is it devstack? | |
| 16:48:49 | fwiesel | fwiesel: On the effort estimate, I can come back to you in a week. | |
| 16:48:51 | bauzas | fwiesel: that's good to hear, that's a first thing | |
| 16:48:52 | grandchild | we have a dedicated qa environment | |
| 16:49:06 | fwiesel | sean-k-mooney: We run it in qa environments on k8s | |
| 16:49:07 | sean-k-mooney | fwiesel: the tird party ci would have to deploy the gerrit chagne under review and report back with tempest test using the vmware driver | |
| 16:49:46 | sean-k-mooney | you would not need to use devstack you could use your k8s install tool provided it uses the nova commit form the review | |
| 16:49:46 | bauzas | fwiesel: do you already have ideas on how many runs you need to do for nova if you add a 3rd party CI ? | |
| 16:49:55 | bauzas | fwiesel: because this can be large | |
| 16:50:37 | sean-k-mooney | bauzas: its much less then it used ot be but yes its non trivial | |
| 16:50:44 | bauzas | https://zuul.openstack.org/builds?project=openstack%2Fnova&branch=master&pipeline=check&skip=0 | |
| 16:50:45 | fwiesel | sean-k-mooney: The main concern is here more that we have to run arbitrary code, so we want it a bit more isolated. | |
| 16:51:16 | sean-k-mooney | fwiesel: yep form past expeince runing a third party ci for nova/neutron | |
| 16:51:24 | sean-k-mooney | geting it approval to run it is non trivial | |
| 16:51:30 | bauzas | fwiesel: grandchild: you can see in the above link how many builds we run | |
| 16:51:38 | bauzas | for the 'check' pipeline | |
| 16:52:33 | bauzas | and we can't unfortunately only test changes that modify the virt driver | |
| 16:52:40 | grandchild | (zuul link throws 500 in fetching the jobs ;) ) | |
| 16:53:17 | bauzas | because we could regress on vmwareapi by a change not related to the driver | |
| 16:53:22 | grandchild | but i'd wager, like fwiesel said, isolation is more of an issue than load. | |
| 16:53:35 | grandchild | bauzas: sure, we'd run the whole pipeline, we are aware of that | |
| 16:53:41 | bauzas | for example, say someone modifies the virt API and forgets to modify vmwareapi module | |
| 16:54:06 | bauzas | grandchild: cool, I just wanted to be aware | |
| 16:54:07 | sean-k-mooney | grandchild: well you dont need to run the full pipeline | |
| 16:54:21 | sean-k-mooney | you only need 1 job that runs on all changes to a subet of direcotries | |
| 16:54:52 | sean-k-mooney | ideally any change the touches the compute and virt modules | |
| 16:54:54 | bauzas | sean-k-mooney: that's my point | |
| 16:54:58 | sean-k-mooney | and ideally som others | |
| 16:54:58 | grandchild | sean-k-mooney: sorry if the terminology is off -- i meant the check for the whole codebase | |
| 16:55:13 | bauzas | sean-k-mooney: I don't really know which modules they should only check | |
| 16:55:47 | bauzas | but we could try to run something small first | |
| 16:55:48 | sean-k-mooney | well ideally every change but at least every chagne to the compute and virt modules | |
| 16:55:55 | bauzas | without removing the experimental tag | |
| 16:56:09 | bauzas | and see how this goes first | |
| 16:56:23 | bauzas | before adding more checks to the 3rd party pipeline | |
| 16:56:37 | sean-k-mooney | sof feature freeze is feb 29th | |
| 16:56:50 | sean-k-mooney | i had wanted the removals to happen before m1 | |
| 16:57:07 | sean-k-mooney | i could be convinced to wait till around m2 for ci or very early febuary | |
| 16:57:16 | grandchild | that sounds fair actually, 3 months to get something on its feet. might be achievable. | |
| 16:57:31 | bauzas | here is what I can propose | |
| 16:57:40 | grandchild | but like fwiesel said, a week for an estimate from us. | |
| 16:57:42 | gibi | (if we remove the the driver from master the driver is be on the stable branches) | |
| 16:57:42 | fwiesel | We could also turn it the other way around. You remove it, and we re-add it, when you are convinced it is fine. | |
| 16:57:49 | bauzas | grandchild: fwiesel: would you be able to attend our meeting on a weekly basis ? | |
| 16:57:57 | grandchild | bauzas: definitely | |
| 16:58:07 | bauzas | okay, here is my proposal then | |
| 16:58:09 | fwiesel | fwiesel: Yes | |
| 16:58:19 | fwiesel | bauzas: Yes | |
| 16:58:24 | bauzas | we could defer the removal until caracal-2 | |
| 16:58:50 | bauzas | and every week, I'd add a topic in our meeting about your 3rd party CI | |
| 16:59:07 | bauzas | you could then tell us where you are and what you need | |
| 16:59:51 | bauzas | and if before end of December, we can't really see anything coming, then we would remove the virt driver | |
| 17:00:04 | fwiesel | Fair enough. | |
| 17:00:05 | grandchild | that sounds fine to me, fwiesel? | |
| 17:00:10 | grandchild | k | |
| 17:00:19 | bauzas | any other opinions but me ? | |
| 17:00:32 | gibi | sound OK to me too | |
| 17:00:33 | bauzas | I'm just a code herder, I'm not a L | |
| 17:00:44 | sean-k-mooney | i can defer writing the remaining patches until after milestone 2 | |
| 17:00:53 | sean-k-mooney | i have the patch to remove the driver already done | |
| 17:01:13 | sean-k-mooney | but i can wait for the api/object changes until we see if the ci turns up by milestone 2 | |
| 17:01:18 | bauzas | okay, then I can write the action item | |
| 17:01:35 | bauzas | sean-k-mooney: I haven't checked, but surely the main patch isn't merged yet, right ? | |
| 17:01:44 | sean-k-mooney | ill -w the removal patch for now | |
| 17:01:49 | sean-k-mooney | its not but i wanted it to be | |
| 17:01:54 | bauzas | okay, sounds a plan | |
| 17:01:58 | sean-k-mooney | i was goign to ask for review today | |
| 17:02:09 | bauzas | and we'll see on a weekly basis how this goes | |
| 17:02:16 | bauzas | lemme write the action | |
| 17:02:53 | bauzas | #agreed we defer the vmwareapi driver removal until caracal-2 | |
| 17:03:29 | bauzas | #agreed fwiesel and/or grandchild will report on a weekly basis about their efforts on running a 3rd party CI for vmwareapi | |
| 17:03:53 | bauzas | #action bauzas to create a meeting topic in order to discuss the efforts | |
| 17:03:55 | grandchild | ftr: jkulik is not in the office today, but also in the mix from us :) | |
| 17:04:03 | bauzas | sure | |