| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-29 | |||
| 13:49:48 | openstackgerrit | Merged openstack/nova master: rbd: use MAX_AVAIL stat for reporting bytes available https://review.opendev.org/556692 | |
| 13:52:09 | openstackgerrit | Merged openstack/nova master: Switch some GitHub URLs to point to opendev.org https://review.opendev.org/677185 | |
| 13:52:17 | openstackgerrit | Merged openstack/nova stable/stein: Add useful error log when _determine_version_cap raises DBNotAllowed https://review.opendev.org/675714 | |
| 13:53:11 | mriedem | Sundar: fair enough, i just looked through the series and the failed tests and indeed they look unrelated, just need rechecking | |
| 13:53:30 | mriedem | as for review, i didn't know it was ready yet - i thought most of the work was being done on the cyborg side (i see the series there isn't merged yet either) | |
| 13:53:54 | donnyd | efried: so it looks like changing consecutive_build_service_disable_threshold=0 solved the issues with ghost disabling of the FN hypervisors. Now I just need to find the proper weight to associate with build_failure_weight_multiplier | |
| 13:54:19 | mriedem | Sundar: it might be good to post something short to the mailing list given a status update, e.g. where the nova patches are (what's ready, what's not - known limitations), and what's going on on the cyborg side - and what the status is on getting a fake cyborg driver for integration testing with tempest going so we can have more confidence in the nova patches | |
| 13:55:07 | mriedem | donnyd: i don't think you do if you set consecutive_build_service_disable_threshold=0 | |
| 13:55:17 | mriedem | 0 * any weight multiplier is 0 | |
| 13:55:47 | donnyd | mriedem: I am going to set that back to an appropriate value, was just making sure that was the reason the hypervisors were being ghost disabled | |
| 13:56:31 | donnyd | I don't want actually broken hypervisors to be in system if nova will take care of that for me | |
| 13:57:19 | donnyd | But i am thinking changing it back to something reasonable, and then setting a weight that doesn't take it out of system for too long would be a good resolution | |
| 13:57:54 | donnyd | but I also may not fully understand how build_failure_weight_multiplier works... so I am going to science the crap out of it | |
| 13:58:45 | mriedem | ♪ she blinded me with science ♪ | |
| 14:02:52 | Sundar | mriedem: Yes, but we are able to test with the Cyborg patches. We will merge them soon. | |
| 14:03:00 | artom | Huh, looks like the NUMA functional tests never actually used the NUMATopologyFilter... | |
| 14:03:02 | bauzas | stephenfin: I'm pretty done with the first patch but I'm confused about one last bit | |
| 14:03:10 | stephenfin | hit me | |
| 14:03:16 | artom | At least, based on the debug logs I'm seeing, that filter's not in there... | |
| 14:03:49 | artom | Err | |
| 14:03:55 | artom | Ignore me :( | |
| 14:03:56 | bauzas | stephenfin: after looking and looking the spec, can't say whether VCPU resources can potentially change or not based on the flags | |
| 14:04:07 | mriedem | artom: see ♪ | |
| 14:04:09 | mriedem | damn | |
| 14:04:13 | mriedem | NUMAServersTestBase | |
| 14:04:22 | mriedem | group='filter_scheduler') | |
| 14:04:22 | mriedem | ['NUMATopologyFilter'], | |
| 14:04:22 | mriedem | self.flags(enabled_filters=CONF.filter_scheduler.enabled_filters + | |
| 14:04:32 | artom | mriedem, right, I know the code's there, I just thought it didn't work for some reason | |
| 14:04:52 | mriedem | throw a raise Exception('does this work?') into the filter and see | |
| 14:04:54 | stephenfin | artom: They definitely use the filter. We've got checks on its side effects | |
| 14:04:56 | mriedem | professional debugging | |
| 14:05:07 | artom | So it's just my func test then :( | |
| 14:05:31 | bauzas | stephenfin: previously, we were only reporting a total VCPUs | |
| 14:05:41 | bauzas | stephenfin: now, we're about to report both VCPU and PCPU | |
| 14:05:51 | bauzas | based on the flags use | |
| 14:06:10 | stephenfin | bauzas: So if I'm understanding you correctly... | |
| 14:06:13 | stephenfin | Previously: | |
| 14:06:36 | stephenfin | no CPU-related configuration -> report all host CPUs (online and offline) as VCPU | |
| 14:06:53 | stephenfin | 'vcpu_pin_set' -> report all host CPUs listed in this option as VCPU | |
| 14:06:54 | bauzas | correct | |
| 14:07:12 | stephenfin | (and there's a check to make sure everything listed in 'vcpu_pin_set' is online) | |
| 14:07:15 | sean-k-mooney | stephenfin: vcpu_pin_set requried all cpus to be online | |
| 14:07:23 | mriedem | damn melwitt is out today and tomorrow | |
| 14:07:26 | sean-k-mooney | if it they were not you could not start the agent | |
| 14:07:29 | bauzas | and dansmith too | |
| 14:07:33 | stephenfin | sean-k-mooney: Ooof, beaten to the punch ;) | |
| 14:07:34 | bauzas | mriedem: ^ | |
| 14:07:46 | mriedem | dan is out tomorrow | |
| 14:07:49 | stephenfin | After: | |
| 14:07:51 | mriedem | i thought anyway | |
| 14:07:55 | sean-k-mooney | stephenfin: :) | |
| 14:08:12 | bauzas | gosh, those people taking perpetual PTOs... | |
| 14:08:12 | stephenfin | no CPU-related configuration -> report all *online* host CPUs as VCPU | |
| 14:08:30 | spatel | sean-k-mooney: ^^ | |
| 14:08:37 | spatel | I need your 5 min | |
| 14:08:51 | bauzas | stephenfin: so, this is a non-breaking change for no CPU-related config, I agree | |
| 14:09:02 | stephenfin | (we now only report the online ones since it was a bug to report offline ones) | |
| 14:09:10 | stephenfin | 'vcpu_pin_set' -> report all host CPUs listed in this option as VCPU | |
| 14:09:11 | bauzas | I agree | |
| 14:09:20 | bauzas | stephenfin: that said, a reno file would be good | |
| 14:09:27 | stephenfin | I thought I had one? | |
| 14:09:53 | stephenfin | https://review.opendev.org/#/c/671793/16/releasenotes/notes/cpu-resources-d4e6a0c12681fa87.yaml ? | |
| 14:10:00 | mriedem | lyarwood: while you're doing stable reviews, can you hit this rocky regression? https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/rocky+topic:bug/1839560 | |
| 14:10:04 | bauzas | stephenfin: sorry, I meant some notes saying we now only report 'active' CPUs | |
| 14:10:10 | stephenfin | ohh, yeah | |
| 14:10:14 | sean-k-mooney | spatel: im just getting back from lunch was about to make coffee whats the issue? | |
| 14:10:34 | stephenfin | bauzas: alex_xu and I discussed that earlier. I said I'd add a follow-up reno, yeah | |
| 14:10:34 | bauzas | anyway, back to the interesting case | |
| 14:10:45 | stephenfin | Was considering making it a bug but we're not going to backport this so meh | |
| 14:10:47 | bauzas | cool then, just adding a comment while we speak on it | |
| 14:10:49 | stephenfin | *filing a bug | |
| 14:10:53 | stephenfin | ack | |
| 14:12:05 | bauzas | anyway, back to the case where people were playing with vcpu_pin_set and they were getting VCPUs | |
| 14:12:16 | bauzas | now, we're deprecating this option | |
| 14:12:20 | artom | mriedem, addressed your feedback in https://review.opendev.org/#/c/635669/ btw, Zuul failure doesn't look related, despite being in the live migration job | |
| 14:12:44 | bauzas | stephenfin: deprecating it is good and that's what we agreed, but we should still report VCPUs, am I right? | |
| 14:12:53 | mriedem | artom: have you looked into why https://review.opendev.org/#/c/678887/13 is failing? | |
| 14:13:06 | Roamer` | mriedem, thanks for the +2 on the StorPool volume driver's native I/O change. Is there a chance it might get a +W too, or should I bother somebody else, or is it too late in the release cycle or something? it's a practically no-risk change (well, okay, obviously you'd have to trust us on that :)) | |
| 14:13:22 | mriedem | Roamer`: takes 2 +2s so you have to find someone else | |
| 14:13:29 | Roamer` | mriedem, thanks | |
| 14:13:38 | spatel | sean-k-mooney: I have VM-1 now i want to build VM-2 on other compute nodes and want to do IP takeover | |
| 14:13:40 | mriedem | another core please see https://review.opendev.org/#/c/676172/ | |
| 14:13:42 | stephenfin | bauzas: Yup, nothing changes from a placement perspective until you unset vcpu_pin_set and set cpu_dedicated_set or cpu_shared_set | |
| 14:13:44 | sean-k-mooney | bauzas: this bit has chagned in the last few days. if you set it we planned to report both vCPUs and pCPUs | |
| 14:13:51 | lyarwood | mriedem: ack will take a look now | |
| 14:13:52 | artom | mriedem, not yet, looking now | |
| 14:14:04 | bauzas | sean-k-mooney: that would have been a bug then | |
| 14:14:11 | Roamer` | so, yeah, anybody up for reviewing a trivial "add io=native to the StorPool volume driver" change? :) https://review.opendev.org/#/c/676172/ | |
| 14:14:14 | sean-k-mooney | bauzas: what would | |
| 14:14:19 | spatel | I have tried to create fix-ip with neutron but its not letting me do that because of policy issue. | |
| 14:14:24 | bauzas | sean-k-mooney: given people were using vcpu_pin_set for pinning, we should stick with only VCPU in this case | |
| 14:14:40 | sean-k-mooney | bauzas: no that breaks upgrades | |
| 14:14:50 | bauzas | gosh | |
| 14:15:09 | sean-k-mooney | vcpu_pin_set defines the set of host cpu that can be used for pinned or floating vms | |
| 14:15:11 | mriedem | stephenfin: i never saw any operator replies to "More upgrade issues with PCPUs - input wanted" - i wonder if you should re-post that but tag with [nova] ? | |
| 14:15:17 | bauzas | sean-k-mooney: what breaks upgrades is having less VCPU between two releases with exact same config options | |
| 14:15:27 | stephenfin | sean-k-mooney: I think you're getting mixed up here | |
| 14:15:30 | mriedem | people, including myself, filter non-tagged things into a general folder and might have missed it b/c of all the noise | |
| 14:15:45 | stephenfin | mriedem: Good call. I need to reply with my resolution for that anyway | |