| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-08-17 | |||
| 17:04:58 | sean-k-mooney | correct | |
| 17:05:06 | sean-k-mooney | on the compute node it calls consume_requests | |
| 17:05:20 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L226-L269 | |
| 17:05:21 | gibi | btw, https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L645 is broken as it individually matches requests to pools and does not consume pools by requests | |
| 17:05:47 | sean-k-mooney | but fixing it in _filter_pools is what i ment by fixing it in one place to fix both | |
| 17:05:48 | gibi | the scheduler calls apply_requests that detects the double consumption | |
| 17:06:14 | gibi | sean-k-mooney: fixing it in filter_pools is tricky as filter_pools does not meant to consume things | |
| 17:06:26 | gibi | can be done though if needed | |
| 17:06:33 | sean-k-mooney | right but apply is not ment to be called in the sculer | |
| 17:06:43 | sean-k-mooney | it will work but we need to make sure not to commit to it on the db | |
| 17:07:00 | sean-k-mooney | so it will need to be a deep copy | |
| 17:07:05 | sean-k-mooney | with no save | |
| 17:07:31 | sean-k-mooney | ok it does not save | |
| 17:07:33 | gibi | nova.scheduler.manager.SchedulerManager._consume_selected_host | |
| 17:07:40 | sean-k-mooney | so as long as we do a deep copy of the pool we are good | |
| 17:07:51 | gibi | that is the one calling apply | |
| 17:08:07 | sean-k-mooney | really for multi create | |
| 17:08:14 | sean-k-mooney | we might already be using a copy by the way | |
| 17:08:19 | gibi | yes probably | |
| 17:08:22 | sean-k-mooney | we had to fix this in the past | |
| 17:15:00 | sean-k-mooney | gibi: https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/host_manager.py#L309 | |
| 17:15:20 | gibi | yepp that one | |
| 17:15:39 | gibi | as you said we need to consume in the scheduler to handle multi create | |
| 17:15:51 | sean-k-mooney | ya ok | |
| 17:16:03 | sean-k-mooney | the host manager has a copy of the pci stats | |
| 17:16:21 | sean-k-mooney | so i guess doing the apply in teh filter might be ok | |
| 17:16:40 | sean-k-mooney | i would have expect _locked_consume_from_request | |
| 17:16:46 | sean-k-mooney | to fail | |
| 17:16:57 | sean-k-mooney | if calling apply_requests was enough | |
| 17:17:02 | gibi | it fails | |
| 17:17:13 | sean-k-mooney | oh so we never get to the compute | |
| 17:17:18 | gibi | but @set_update_time_on_success catches everything | |
| 17:17:29 | sean-k-mooney | i tought you said we got to the compute | |
| 17:17:38 | gibi | https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/host_manager.py#L266 | |
| 17:17:50 | gibi | https://opendev.org/openstack/nova/src/branch/master/nova/scheduler/host_manager.py#L81 | |
| 17:18:03 | gibi | this is where scheduler points to the compute :D | |
| 17:18:34 | sean-k-mooney | ok but we dont actully select the host and proced to we | |
| 17:18:50 | sean-k-mooney | i guess we might if noting depend on teh result of the consume | |
| 17:19:05 | sean-k-mooney | ok it returns nothing | |
| 17:19:11 | sean-k-mooney | and since that just logs | |
| 17:19:14 | sean-k-mooney | we continue and fail | |
| 17:19:16 | sean-k-mooney | on the compute | |
| 17:19:17 | gibi | yepp that just logs | |
| 17:19:30 | sean-k-mooney | ok so that should raise an instance build excption or similr | |
| 17:19:47 | sean-k-mooney | so we retry the next alternit host | |
| 17:20:02 | sean-k-mooney | well no | |
| 17:20:08 | sean-k-mooney | we need to fail in the filter | |
| 17:20:13 | sean-k-mooney | so call apply there | |
| 17:20:38 | sean-k-mooney | but we proably should aslo adress the fact that eats valid excptions | |
| 17:21:19 | gibi | and this is where compute detects the error and only logs, does not fail the instance build https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L244 | |
| 17:21:29 | gibi | it point to the scheduler :D | |
| 17:21:46 | sean-k-mooney | well its ment to return none | |
| 17:21:58 | sean-k-mooney | and the caller shoudl error but i guess it does not | |
| 17:22:31 | sean-k-mooney | i.e. the caller proably shoudl be checkign the lenght of the allocation to the requests | |
| 17:22:33 | gibi | also apply does not care about dependent devices, but consume_requests does, so we might actually need to call consume_requests | |
| 17:22:59 | sean-k-mooney | so suports request only exists to not call consume | |
| 17:23:09 | sean-k-mooney | but we proably could remvoe it can call consume | |
| 17:23:17 | gibi | yeah probably | |
| 17:23:35 | sean-k-mooney | the reason we dont right now | |
| 17:23:57 | sean-k-mooney | is incase the host is eliminated by another filter | |
| 17:24:18 | sean-k-mooney | we want to make sure the host state object has the correct data | |
| 17:24:40 | sean-k-mooney | but we can do that by doing a deep copy of the pci stats in the filter | |
| 17:25:02 | sean-k-mooney | there are other ways around that too | |
| 17:25:21 | sean-k-mooney | this comes back to correct behavior for muticreate/server groups | |
| 17:25:57 | sean-k-mooney | i would proably try just calling consume and see if that breaks any of our tests | |
| 17:26:10 | sean-k-mooney | then we might eb able to remove the call to apply | |
| 17:26:13 | sean-k-mooney | in the host manager | |
| 17:26:16 | gibi | yepp | |
| 17:26:20 | gibi | that make sense | |
| 17:40:11 | spatel | can i set hard limit in nova to create number of vm instance? | |
| 18:15:00 | JayF | So, trying to figure out why gate jobs keep failing on my change. They are seemingly random, and I noticed something strange in zuul status: https://home.jvf.cc/~jay/gated-after-master-change.png | |
| 18:15:18 | JayF | 826523,5 is a change landing into master AFAICT | |
| 18:15:28 | JayF | but is in the same merge queue as my change to stable/yoga (800873,2) | |
| 18:16:05 | JayF | s/yoga/victoria/ | |
| 18:17:33 | sean-k-mooney | they might be intermiten failure yes | |
| 18:17:52 | JayF | Well, I'm just saying I'm surprised that stable/ patches and master patches are gate-queued together | |
| 18:18:11 | sean-k-mooney | oh they should not be unless its tempest | |
| 18:18:23 | JayF | that's what my png ^^^ shows | |
| 18:18:34 | JayF | is that it seems to have done exactly that | |
| 18:18:52 | sean-k-mooney | do you have the link to the backprot | |
| 18:18:55 | JayF | oooh, or do the separate lines indicate "nope, this is in a separate queue" | |
| 18:19:06 | JayF | https://review.opendev.org/c/openstack/nova/+/800873 same as yesterday | |
| 18:19:09 | JayF | trying to get it to actually land | |
| 18:19:23 | sean-k-mooney | the seperate lines are seperate sets of patches that are in flight | |
| 18:19:37 | sean-k-mooney | so those cidner changes are sperate form the nova one | |
| 18:19:54 | JayF | got it, this is just me learning for the first time after looking at this UI for an embarassingly long time | |
| 18:19:59 | JayF | that the lines on the left have meaning | |
| 18:20:02 | JayF | lol | |
| 18:20:05 | JayF | thank you for the clarity :D | |
| 18:20:30 | sean-k-mooney | there is somethign wrong here | |
| 18:20:50 | sean-k-mooney | well hum im not sure | |
| 18:20:58 | sean-k-mooney | 800873,2 is a master patch | |
| 18:21:10 | sean-k-mooney | but i dont think it depends on the 826523,5 | |
| 18:21:18 | JayF | https://review.opendev.org/c/openstack/nova/+/800873 that's not true | |
| 18:21:20 | sean-k-mooney | i think the patch it was specultivly merged with has already merged | |
| 18:21:37 | JayF | this says repo | branch: openstack/nova | stable/victoria | |
| 18:21:48 | sean-k-mooney | on that yes | |
| 18:21:53 | sean-k-mooney | but not on https://review.opendev.org/c/openstack/nova/+/826523/ | |
| 18:21:58 | JayF | that is 800873,2 | |
| 18:22:06 | sean-k-mooney | that is the second nova patch with the x | |