| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-04 | |||
| 18:11:14 | sean-k-mooney | dansmith: regarding ci resouces. one thing that i have tought about from time to time was spliting check into check and fast check with check dependt on fast check | |
| 18:11:40 | sean-k-mooney | now we said we dont want to make them dependent because we want all the result at once | |
| 18:12:07 | sean-k-mooney | but if we had two pipelines liek that we might be abel to run only the logner jobs if the patch did not have -w | |
| 18:12:21 | sean-k-mooney | im not sure if that would save much resouces | |
| 18:12:49 | sean-k-mooney | but sometimes im torn between pushing code to gerrit to have a backup or make shareing ti between multiple servers simpler | |
| 18:12:55 | sean-k-mooney | and wasting gate resouces | |
| 18:13:21 | sean-k-mooney | if we had a way to say dont run the jobs yet that might help with early verions | |
| 18:14:25 | sean-k-mooney | in the grand scheme of things its proably not going to be large but it might be worth exploring having a ready-for-ci lable or something | |
| 18:16:07 | sean-k-mooney | i did that for my third party ci https://github.com/SeanMooney/ci-sean-mooney/blob/main/zuul.d/pipelines.yaml#L53-L56 | |
| 18:16:27 | sean-k-mooney | and its how the intel nfv ci used to run to save capsity | |
| 18:16:58 | sean-k-mooney | althogh it was not night and day or anything | |
| 18:29:02 | dansmith | sean-k-mooney: yeah I'm not sure if that's really doable, but I would love to get *some* results before others, that would help a lot | |
| 18:29:32 | sean-k-mooney | what i suggested before was ll the non tempest ones first then the rest | |
| 18:29:48 | sean-k-mooney | granted you can run those simpley locally | |
| 18:31:18 | sean-k-mooney | its too bad zuul cant report back as each job finishes but when i realy want that i do go to zuul.openstack.org | |
| 18:31:31 | sean-k-mooney | and just get teh results from there instead of waiting | |
| 18:32:16 | sean-k-mooney | the results are avaiable in zuul once the indivuatl job finsihes just not in gerrit | |
| 18:46:54 | dansmith | sean-k-mooney: yeah I'd like to have one tempest job and the easy ones in the first go I think | |
| 18:47:04 | dansmith | worker counts being lower would make that still go faster I think | |
| 18:47:14 | sean-k-mooney | yep you could do that | |
| 18:47:26 | sean-k-mooney | choose one of the faster ones | |
| 18:47:32 | dansmith | but I think zuul lacks some persistence required to split up the job and still know when it can gate | |
| 18:47:33 | dansmith | so not sure that's really an option | |
| 18:47:47 | sean-k-mooney | im not sure about that | |
| 18:47:58 | sean-k-mooney | it curerntly does it based on lables | |
| 18:48:05 | dansmith | using something like experimental and requiring a +1 experimental run before gate would be a hack around that maybe | |
| 18:48:16 | dansmith | sean-k-mooney: well, talk to the infra folks, but my understanding is it's hard | |
| 18:48:46 | sean-k-mooney | we woudl jsut need a requires claus in the gate piple to look for verifed and fast-verfied +1 from zuul | |
| 18:49:00 | dansmith | in addition to solving this by dividing up the problem or saying "zuul should have a feature" I think there is a LOT of work we all can do to make things faster, duplicate less, and be more targetd | |
| 18:49:51 | sean-k-mooney | currently its looking for just verifed and workflow https://github.com/openstack/project-config/blob/master/zuul.d/pipelines.yaml#L80-L82 | |
| 18:50:24 | sean-k-mooney | but you could add a 3rd labple to that e.g. fast-verifed and still requrie a +1 form both form zuul | |
| 18:50:58 | dansmith | okay but without a prioritization, we'd still be hours and hours before running | |
| 18:50:59 | sean-k-mooney | dansmith: it would require use to update the gerrit config and add the feature however | |
| 18:51:19 | dansmith | I have stuff that has been in the check queue for three hours and it hasn't started to run a single thing | |
| 18:51:19 | sean-k-mooney | we have precedence: normal | |
| 18:51:30 | sean-k-mooney | for prioritisation beteween pipliens | |
| 18:51:38 | dansmith | I've already talked to infra about this, | |
| 18:51:43 | dansmith | and the other precedences are used for things | |
| 18:51:49 | dansmith | check and experimental are the same even | |
| 18:51:50 | sean-k-mooney | yep they are | |
| 18:51:58 | sean-k-mooney | yep both low | |
| 18:52:07 | sean-k-mooney | althoguh experimtal will report back first | |
| 18:52:20 | dansmith | again, I think we can do a lot without making this an infra problem | |
| 18:52:25 | sean-k-mooney | they have teh same precidence but are in differnet queues | |
| 18:52:33 | dansmith | and just making it so we get fast check in two hours and slow check in 24 hours isn't really going to help | |
| 18:52:33 | sean-k-mooney | dansmith: oh ya i know | |
| 18:52:47 | sean-k-mooney | its more if we run out of room with your current effort | |
| 18:52:54 | sean-k-mooney | there are other things we can do with infra | |
| 18:52:57 | sean-k-mooney | but its more involed | |
| 18:53:46 | sean-k-mooney | im not suggesting we start with infra changes just pointing out we can do things via infra changes if its still a proablem | |
| 18:53:47 | dansmith | there's lots we could ask infra to do, but relative to the staffing of the top five projects, I mean.. :) | |
| 18:54:34 | sean-k-mooney | the other thing too is you were just looking at 1st party ci | |
| 18:54:36 | dansmith | I get the impression the things we "could do with infra" require a very wide-scope of potential considerations, more than we can just do in our job defs, and likely would need zuul changes | |
| 18:54:59 | sean-k-mooney | dansmith: yes the infra chagnes are openstack wide | |
| 18:55:08 | sean-k-mooney | requireign both gerrit and zull configuration chagnes | |
| 18:55:17 | sean-k-mooney | so very big/wide reaching hammer | |
| 18:55:48 | sean-k-mooney | not running 8 almost identicaly jobs is relitivly local in contrast | |
| 18:56:01 | dansmith | yeah | |
| 18:56:32 | dansmith | I wish I could help accelerate us not running the two grenades because those are fairly heavy and really duplicative | |
| 18:56:51 | sean-k-mooney | well we can stop that in nova | |
| 18:57:02 | sean-k-mooney | right now if we want too | |
| 18:57:16 | dansmith | I know, but I think we agreed to wait until the ceph and zuulv3 thing was resolved | |
| 18:57:25 | dansmith | I already proposed it with -W to wait on that | |
| 18:58:44 | sean-k-mooney | well mor i ment we can drop integrated-gate-compute template then contol which grenddade jobs run rom the check and gate pipelines | |
| 18:58:51 | sean-k-mooney | https://github.com/openstack/nova/blob/master/.zuul.yaml#L421 | |
| 18:59:04 | sean-k-mooney | which means we coudl jsut run - nova-grenade-multinode | |
| 18:59:21 | dansmith | https://review.opendev.org/c/openstack/tempest/+/771499 | |
| 18:59:43 | dansmith | we agreed we would wait to do that until the ceph multinode zuulv3 thing was resolved | |
| 19:00:15 | sean-k-mooney | oh i know i just was pointing out we could do it via nova if we had resovled it and not need a tempest patch | |
| 19:01:02 | dansmith | and when we discussed, gmann wanted it changed there ^, but yes, the mechanics aren't hard, it's the agreement required, and in this case, blocking on the zuulv3 conversion | |
| 19:01:26 | alexe9191 | good day everyone:) | |
| 19:01:34 | sean-k-mooney | alexe9191: o/ | |
| 19:01:35 | alexe9191 | I have a question about the retry filter in nova | |
| 19:01:50 | alexe9191 | I am wondering where does it get it's spec_obj from ? specefically this piece of code here: | |
| 19:01:50 | alexe9191 | def host_passes(self, host_state, spec_obj): | |
| 19:01:51 | alexe9191 | """Skip nodes that have already been attempted.""" | |
| 19:01:51 | alexe9191 | retry = spec_obj.retry | |
| 19:01:52 | sean-k-mooney | alexe9191: it has not been required for quite some time | |
| 19:02:24 | sean-k-mooney | alexe9191: its passed in by the filter schduler | |
| 19:02:33 | alexe9191 | but where is it stored? memory or db? | |
| 19:02:58 | sean-k-mooney | its the request spec | |
| 19:03:04 | sean-k-mooney | its builts in the api | |
| 19:03:12 | sean-k-mooney | then passed to the conductor and scudler | |
| 19:03:22 | sean-k-mooney | i belive we might have it in the api db | |
| 19:03:35 | alexe9191 | Interesting, let me check | |
| 19:03:43 | dansmith | yes, api_db | |
| 19:04:30 | alexe9191 | ok, so if a host fail, it will register it's state here in that table. | |
| 19:04:32 | alexe9191 | request_specs | |
| 19:04:45 | sean-k-mooney | no | |
| 19:04:57 | sean-k-mooney | we dont commit that to the db | |
| 19:05:21 | sean-k-mooney | we only track that in memroy i belive | |
| 19:05:31 | alexe9191 | that table is quite loaded though. | |
| 19:05:47 | sean-k-mooney | the request_sepc is used for other things | |
| 19:05:53 | sean-k-mooney | so it is saved in the db | |
| 19:06:11 | sean-k-mooney | but wee dont save the failed host in the request spec during schdule and commit that back | |
| 19:06:30 | alexe9191 | so that part right here: | |
| 19:06:31 | alexe9191 | return self.filter_handler.get_filtered_objects(self.enabled_filters, | |
| 19:06:31 | alexe9191 | hosts, spec_obj, index) | |
| 19:07:02 | alexe9191 | i see it's passing hosts but the retry filter is kicking out all of the hosts in the aggregate I am trying to schedule in. They are all healthy and they are hosting VMs, they probably had issues at some time. | |
| 19:07:21 | alexe9191 | The interesting thing, this happens only with one flavor, other flavors are returning a different results for the retryfilter. | |