| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-30 | |||
| 14:12:19 | bauzas | if executed in separate workers, CONF.cpu_allocation_ratio wouldn't be defined for nova-scheduler | |
| 14:12:36 | bauzas | oh wait, sec | |
| 14:12:51 | mriedem | it doesn't need to be defined in config, we have a default | |
| 14:12:53 | mriedem | which is global | |
| 14:13:17 | mriedem | anyway, that doesn't really matter for this bug | |
| 14:13:28 | mriedem | the compute reports the inventory to placement and is reporting 0.0 allocation ratios | |
| 14:14:20 | cdent | mriedem: one thing that remains unclear for me (becuase apparently I can't read python code) is why the second inventory PUT (the one with the 0.0) is happening at all (and so soon) | |
| 14:18:03 | bauzas | mriedem: I think your working theory you stated in the bug comment is valid | |
| 14:18:22 | bauzas | I'm trying to wrap my head around on exactly when we pull the DB values | |
| 14:19:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Update resources once in update_available_resource" https://review.openstack.org/598176 | |
| 14:21:12 | bauzas | mriedem: I think we began having problems with https://github.com/openstack/nova/commit/7b95b4d60726b6c8d0e0fe939c408a91ada79e0c | |
| 14:21:36 | bauzas | I'm not saying it's all the bugs cause | |
| 14:21:57 | bauzas | just that it implicitly creates a dependency on us calling .save() after it | |
| 14:22:10 | bauzas | because if not, we would then have 0.0 values | |
| 14:23:48 | mriedem | well, it seems pretty obvious to me that we just shouldn't set 0.0 values on the compute node record in _copy_resources if the config is still just 0.0 | |
| 14:24:31 | bauzas | mriedem: sure, we could leave the defaults | |
| 14:24:44 | bauzas | because once we pull the object, the facade fixes it for free | |
| 14:27:01 | mriedem | except we're no longer pulling the object in the RT | |
| 14:27:09 | mriedem | which is i think the problem | |
| 14:27:58 | zigo | mriedem: Sure, let me add it. | |
| 14:28:24 | zigo | mriedem: Yes, I package tags. | |
| 14:29:06 | mriedem | zigo: you don't want to add it | |
| 14:29:24 | bauzas | mriedem: sorry, you mean we're getting the CPU, RAM and disk values out of the DB directly ? | |
| 14:29:39 | bauzas | I thought we were still getting by id ? | |
| 14:29:42 | mriedem | zigo: i'm asking because https://review.openstack.org/#/c/520024/ is in master only, not rocky, but it looks like the culprit of the failure - but that wouldn't then explain how you'd be failing in rocky | |
| 14:30:17 | bauzas | me looks at https://github.com/openstack/nova/blob/f534495a427d1683bc536cf003ec02edbf6d8a45/nova/compute/resource_tracker.py#L85 | |
| 14:31:35 | zigo | mriedem: I have rc2 packaged, not rc3. | |
| 14:31:38 | zigo | Maybe I should update? | |
| 14:32:08 | mriedem | yeah probably | |
| 14:32:13 | mriedem | or just the GA | |
| 14:32:16 | mriedem | which was released today? | |
| 14:32:36 | mriedem | https://github.com/openstack/nova/tree/18.0.0 | |
| 14:32:59 | mriedem | yeah 20 minutes ago http://git.openstack.org/cgit/openstack/nova/tag/?h=18.0.0 | |
| 14:35:59 | zigo | Ah, right ! | |
| 14:36:01 | zigo | Doing that. | |
| 14:36:21 | cdent | I'm thinking that the positive way of looking at this is that the second _update has been masking a nasty bug for a long time | |
| 14:41:48 | mriedem | oh i'm not surprised that there would be a big pile of tight coupling tape holding this all together | |
| 14:42:05 | mriedem | and why i didn't want to merge that change before the GA | |
| 14:49:43 | bauzas | mriedem: again, the more I read, the more I think we possibly had the original problem once we had https://github.com/openstack/nova/commit/7b95b4d60726b6c8d0e0fe939c408a91ada79e0c | |
| 14:49:53 | bauzas | and .update() was just hiding it | |
| 14:50:12 | bauzas | mriedem: so, do you want me to modify the above and only set the values if not 0.0 ? | |
| 14:50:19 | bauzas | or are you working on this ? | |
| 14:52:34 | bauzas | disclaimer: looking at a long list of openstack-dev threads | |
| 15:00:27 | mriedem | Kevin_Zheng: yikun: you might be interested in https://review.openstack.org/#/c/591976/ | |
| 15:00:47 | mriedem | bauzas: i'm working it | |
| 15:00:59 | bauzas | okay | |
| 15:01:21 | Kevin_Zheng | Got it | |
| 15:02:31 | zigo | Nova 18.0.0 built, doing a recheck... | |
| 15:03:01 | openstackgerrit | Merged openstack/nova master: Fix soft deleting vm fails after "nova resize" vm https://review.openstack.org/546920 | |
| 15:04:36 | melwitt | . | |
| 15:16:00 | melwitt | stephenfin: on https://review.openstack.org/595592, if the bug is new for rocky, how did moshele run into it on OSP13 (queens)? | |
| 15:16:24 | stephenfin | melwitt: If he's using OSP, then it's because we backported it downstream | |
| 15:16:38 | stephenfin | Even if not, I'd imagine it's a feature backport | |
| 15:18:17 | melwitt | stephenfin: ohh, ok. I didn't think of that, backport of a feature | |
| 15:30:40 | mriedem | ? | |
| 15:30:40 | mriedem | efried: does any kind of nova/cyborg integration actually exist to do anything with servers in rocky/ | |
| 15:31:02 | mriedem | b/c https://docs.openstack.org/releasenotes/cyborg/rocky.html says it does | |
| 15:31:36 | mriedem | 1) rocky release notes talk about queens specs | |
| 15:31:47 | mriedem | 2) it sounds like "we completed specs" rather than "we have functioning code" | |
| 15:32:51 | Kevin_Zheng | mriedem: you made me laugh | |
| 15:33:43 | mriedem | i just don't want people showing up in -nova saying "hey why can't i attach fpgas to my vm?" | |
| 15:38:28 | efried | mriedem: No, cyborg has nothing in nova atm | |
| 15:38:46 | efried | oh, yeah, that's poorly worded. | |
| 15:41:16 | Kevin_Zheng | mriedem: we were talking today, about the batch when listing patch | |
| 15:42:24 | Kevin_Zheng | The batch size could vary depending on sort key and dir | |
| 15:42:44 | Kevin_Zheng | Like for sort with uuid | |
| 15:43:01 | Kevin_Zheng | It could be evenly distributed | |
| 15:43:24 | dansmith | Kevin_Zheng: you mean the *optimal* batch size? | |
| 15:43:40 | Kevin_Zheng | dansmith: yeah | |
| 15:44:40 | dansmith | Kevin_Zheng: obviously if you're sorting by uuid it should be fairly evenly distributes and sorting by other things could be massively less well-distributed, but I'm not sure how we could (efficiently) optimize that at runtime | |
| 15:44:45 | dansmith | Kevin_Zheng: do you have ideas? | |
| 15:45:00 | Kevin_Zheng | I was thinking a tool that analyzes dB data | |
| 15:45:16 | Kevin_Zheng | And feed to nova periodically | |
| 15:45:30 | Kevin_Zheng | But that seems to much :) | |
| 15:45:36 | dansmith | yeah :) | |
| 15:46:08 | dansmith | it likely varies by tenant, sort_key, cloud layout, scheduler weights, etc | |
| 15:46:17 | Kevin_Zheng | Just came up when we introduced the new approach to our product team | |
| 15:46:18 | dansmith | it'd be hard to pin that down except for a single-tenant cloud I think | |
| 15:46:55 | sean-k-mooney | mriedem: today i belive you can use cyborg to program a pci device and then you can use nova to pass it through via a pci passthrough flavor alisa but there is no way to force landing on the host with the device you just programed | |
| 15:46:57 | Kevin_Zheng | I’d say it is a powerful tool:) | |
| 15:47:33 | dansmith | Kevin_Zheng: if the goal is to get larger batches from cells likely to have many results, we could do things like scale up the batch size each time you hit a cell again | |
| 15:48:03 | dansmith | Kevin_Zheng: so that if your query is likely to get most results from one cell, we get $batch_size, then $batch_size*2, then $batch_size*4, etc | |
| 15:48:25 | dansmith | but I think I'd want to see a benchmark showing that as worthwhile before I approved it, | |
| 15:48:34 | Kevin_Zheng | Hmm | |
| 15:48:51 | Kevin_Zheng | That could be a good way | |
| 15:48:51 | dansmith | because I expect that since the db query time is so small compared to the processing time, I'm not sure it matters that much (even your hyper-optimized batch sizing tool :) | |
| 15:49:26 | Kevin_Zheng | Yeah, they are just guessing as always | |
| 15:49:49 | dansmith | Kevin_Zheng: yeah :D | |
| 15:50:27 | dansmith | Kevin_Zheng: it's common trap: One big optimization on batch size gives 60% improvement, so assume there are more 60% improvements to be gained through hyper-optimization :) | |
| 15:50:34 | openstackgerrit | Merged openstack/nova-specs master: VMware: add support for live migration https://review.openstack.org/598163 | |
| 15:59:27 | mriedem | leave some optimizations for the enterprise fellas | |
| 16:01:51 | mriedem | gibi: you were +2 on this before i robustified the test per mel's prodding https://review.openstack.org/#/c/588943/ | |
| 16:03:51 | Kevin_Zheng | maybe left some place for them to be able to do that, like a call to my powerful tool backend :P | |
| 16:04:53 | mriedem | is the toronto lab already working on that? | |
| 16:05:15 | mriedem | research people gotta get grant money somehow | |
| 16:05:37 | sean-k-mooney | mriedem: i will likely be fixing a few things in cyborg in the near future. do you want me to fix the releases note regarding nova inetgration | |
| 16:05:43 | mriedem | need i remind everybody https://www.openstack.org/videos/vancouver-2018/revisiting-scalability-and-applicability-of-openstack-placement-1 | |
| 16:05:58 | mriedem | sean-k-mooney: i guess? | |
| 16:06:05 | mriedem | revising release notes is sometimes a tricky business | |
| 16:06:47 | sean-k-mooney | well we update specs retroactivly i done really see release notes as any different | |