| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-29 | |||
| 13:26:20 | efried | (in -oslo) | |
| 13:30:21 | mriedem | dansmith: i've updated this prep resize at dest cross-cell patch to remove some of the unused stuff and cleanup the commit/docstrings https://review.opendev.org/#/c/633293/46 - i'm still not really sold on munging this into the existing prep_resize and tried to explain why in the commit and docstring on the method if that helps | |
| 13:31:18 | mriedem | i of course can if it's going to be a blocker, i think it would mean changing prep_resize to consider if migration.cross_cell_move is True and if so, don't reschedule and don't cast to resize_instance on the source | |
| 13:34:23 | bauzas | efried: I think we stated a lot of implications between shared and pinned instances in the spec | |
| 13:35:58 | efried | bauzas: Yeah, I just got a poke from elsewhere that made me remember seeing that assumption/limitation in the code somewhere and I was like "wha...?" | |
| 13:36:20 | bauzas | efried: I'd recommend you to open the spec while you review the cpu-resources series | |
| 13:36:29 | bauzas | that and take some aspirin | |
| 13:36:47 | bauzas | I've started reviewing it but I'm not yet able to sharpen my pen | |
| 13:36:50 | efried | yes, I have it (the in-flight edited version) open. | |
| 13:37:24 | efried | I've been thrashing like a 386 running minecraft though, been reviewing like three patches every other day. | |
| 13:42:03 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Move live_migration test hooks under gate/ https://review.opendev.org/679251 | |
| 13:44:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Rename the nova-grenade-live-migration job to nova-grenade-multinode https://review.opendev.org/678841 | |
| 13:44:40 | openstackgerrit | Merged openstack/nova master: Docs and functional test for max_local_block_devices https://review.opendev.org/678268 | |
| 13:44:56 | openstackgerrit | Merged openstack/nova master: Add a "Caveats" section to the eventlet profiling docs https://review.opendev.org/676672 | |
| 13:45:05 | openstackgerrit | Merged openstack/nova master: docs: pretty up return code table or heal_allocations https://review.opendev.org/678063 | |
| 13:45:18 | openstackgerrit | Merged openstack/nova master: doc: pretty up return code table for sync_aggregates https://review.opendev.org/678064 | |
| 13:46:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add cold migrate and resize to nova-grenade-multinode https://review.opendev.org/679210 | |
| 13:47:38 | Sundar | mriedem: Re. the comment in https://etherpad.openstack.org/p/nova-runways-train that tests are failing on Cyborg patches, as of now, they are few and seem unrelated to the code. Even back then, there was really only 1 failure related to the patches. What can we do to get reviews on these patches? | |
| 13:49:48 | openstackgerrit | Merged openstack/nova master: rbd: use MAX_AVAIL stat for reporting bytes available https://review.opendev.org/556692 | |
| 13:52:09 | openstackgerrit | Merged openstack/nova master: Switch some GitHub URLs to point to opendev.org https://review.opendev.org/677185 | |
| 13:52:17 | openstackgerrit | Merged openstack/nova stable/stein: Add useful error log when _determine_version_cap raises DBNotAllowed https://review.opendev.org/675714 | |
| 13:53:11 | mriedem | Sundar: fair enough, i just looked through the series and the failed tests and indeed they look unrelated, just need rechecking | |
| 13:53:30 | mriedem | as for review, i didn't know it was ready yet - i thought most of the work was being done on the cyborg side (i see the series there isn't merged yet either) | |
| 13:53:54 | donnyd | efried: so it looks like changing consecutive_build_service_disable_threshold=0 solved the issues with ghost disabling of the FN hypervisors. Now I just need to find the proper weight to associate with build_failure_weight_multiplier | |
| 13:54:19 | mriedem | Sundar: it might be good to post something short to the mailing list given a status update, e.g. where the nova patches are (what's ready, what's not - known limitations), and what's going on on the cyborg side - and what the status is on getting a fake cyborg driver for integration testing with tempest going so we can have more confidence in the nova patches | |
| 13:55:07 | mriedem | donnyd: i don't think you do if you set consecutive_build_service_disable_threshold=0 | |
| 13:55:17 | mriedem | 0 * any weight multiplier is 0 | |
| 13:55:47 | donnyd | mriedem: I am going to set that back to an appropriate value, was just making sure that was the reason the hypervisors were being ghost disabled | |
| 13:56:31 | donnyd | I don't want actually broken hypervisors to be in system if nova will take care of that for me | |
| 13:57:19 | donnyd | But i am thinking changing it back to something reasonable, and then setting a weight that doesn't take it out of system for too long would be a good resolution | |
| 13:57:54 | donnyd | but I also may not fully understand how build_failure_weight_multiplier works... so I am going to science the crap out of it | |
| 13:58:45 | mriedem | ♪ she blinded me with science ♪ | |
| 14:02:52 | Sundar | mriedem: Yes, but we are able to test with the Cyborg patches. We will merge them soon. | |
| 14:03:00 | artom | Huh, looks like the NUMA functional tests never actually used the NUMATopologyFilter... | |
| 14:03:02 | bauzas | stephenfin: I'm pretty done with the first patch but I'm confused about one last bit | |
| 14:03:10 | stephenfin | hit me | |
| 14:03:16 | artom | At least, based on the debug logs I'm seeing, that filter's not in there... | |
| 14:03:49 | artom | Err | |
| 14:03:55 | artom | Ignore me :( | |
| 14:03:56 | bauzas | stephenfin: after looking and looking the spec, can't say whether VCPU resources can potentially change or not based on the flags | |
| 14:04:07 | mriedem | artom: see ♪ | |
| 14:04:09 | mriedem | damn | |
| 14:04:13 | mriedem | NUMAServersTestBase | |
| 14:04:22 | mriedem | self.flags(enabled_filters=CONF.filter_scheduler.enabled_filters + | |
| 14:04:22 | mriedem | ['NUMATopologyFilter'], | |
| 14:04:22 | mriedem | group='filter_scheduler') | |
| 14:04:32 | artom | mriedem, right, I know the code's there, I just thought it didn't work for some reason | |
| 14:04:52 | mriedem | throw a raise Exception('does this work?') into the filter and see | |
| 14:04:54 | stephenfin | artom: They definitely use the filter. We've got checks on its side effects | |
| 14:04:56 | mriedem | professional debugging | |
| 14:05:07 | artom | So it's just my func test then :( | |
| 14:05:31 | bauzas | stephenfin: previously, we were only reporting a total VCPUs | |
| 14:05:41 | bauzas | stephenfin: now, we're about to report both VCPU and PCPU | |
| 14:05:51 | bauzas | based on the flags use | |
| 14:06:10 | stephenfin | bauzas: So if I'm understanding you correctly... | |
| 14:06:13 | stephenfin | Previously: | |
| 14:06:36 | stephenfin | no CPU-related configuration -> report all host CPUs (online and offline) as VCPU | |
| 14:06:53 | stephenfin | 'vcpu_pin_set' -> report all host CPUs listed in this option as VCPU | |
| 14:06:54 | bauzas | correct | |
| 14:07:12 | stephenfin | (and there's a check to make sure everything listed in 'vcpu_pin_set' is online) | |
| 14:07:15 | sean-k-mooney | stephenfin: vcpu_pin_set requried all cpus to be online | |
| 14:07:23 | mriedem | damn melwitt is out today and tomorrow | |
| 14:07:26 | sean-k-mooney | if it they were not you could not start the agent | |
| 14:07:29 | bauzas | and dansmith too | |
| 14:07:33 | stephenfin | sean-k-mooney: Ooof, beaten to the punch ;) | |
| 14:07:34 | bauzas | mriedem: ^ | |
| 14:07:46 | mriedem | dan is out tomorrow | |
| 14:07:49 | stephenfin | After: | |
| 14:07:51 | mriedem | i thought anyway | |
| 14:07:55 | sean-k-mooney | stephenfin: :) | |
| 14:08:12 | stephenfin | no CPU-related configuration -> report all *online* host CPUs as VCPU | |
| 14:08:12 | bauzas | gosh, those people taking perpetual PTOs... | |
| 14:08:30 | spatel | sean-k-mooney: ^^ | |
| 14:08:37 | spatel | I need your 5 min | |
| 14:08:51 | bauzas | stephenfin: so, this is a non-breaking change for no CPU-related config, I agree | |
| 14:09:02 | stephenfin | (we now only report the online ones since it was a bug to report offline ones) | |
| 14:09:10 | stephenfin | 'vcpu_pin_set' -> report all host CPUs listed in this option as VCPU | |
| 14:09:11 | bauzas | I agree | |
| 14:09:20 | bauzas | stephenfin: that said, a reno file would be good | |
| 14:09:27 | stephenfin | I thought I had one? | |
| 14:09:53 | stephenfin | https://review.opendev.org/#/c/671793/16/releasenotes/notes/cpu-resources-d4e6a0c12681fa87.yaml ? | |
| 14:10:00 | mriedem | lyarwood: while you're doing stable reviews, can you hit this rocky regression? https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/rocky+topic:bug/1839560 | |
| 14:10:04 | bauzas | stephenfin: sorry, I meant some notes saying we now only report 'active' CPUs | |
| 14:10:10 | stephenfin | ohh, yeah | |
| 14:10:14 | sean-k-mooney | spatel: im just getting back from lunch was about to make coffee whats the issue? | |
| 14:10:34 | bauzas | anyway, back to the interesting case | |
| 14:10:34 | stephenfin | bauzas: alex_xu and I discussed that earlier. I said I'd add a follow-up reno, yeah | |
| 14:10:45 | stephenfin | Was considering making it a bug but we're not going to backport this so meh | |
| 14:10:47 | bauzas | cool then, just adding a comment while we speak on it | |
| 14:10:49 | stephenfin | *filing a bug | |
| 14:10:53 | stephenfin | ack | |
| 14:12:05 | bauzas | anyway, back to the case where people were playing with vcpu_pin_set and they were getting VCPUs | |
| 14:12:16 | bauzas | now, we're deprecating this option | |
| 14:12:20 | artom | mriedem, addressed your feedback in https://review.opendev.org/#/c/635669/ btw, Zuul failure doesn't look related, despite being in the live migration job | |
| 14:12:44 | bauzas | stephenfin: deprecating it is good and that's what we agreed, but we should still report VCPUs, am I right? | |
| 14:12:53 | mriedem | artom: have you looked into why https://review.opendev.org/#/c/678887/13 is failing? | |
| 14:13:06 | Roamer` | mriedem, thanks for the +2 on the StorPool volume driver's native I/O change. Is there a chance it might get a +W too, or should I bother somebody else, or is it too late in the release cycle or something? it's a practically no-risk change (well, okay, obviously you'd have to trust us on that :)) | |
| 14:13:22 | mriedem | Roamer`: takes 2 +2s so you have to find someone else | |
| 14:13:29 | Roamer` | mriedem, thanks | |
| 14:13:38 | spatel | sean-k-mooney: I have VM-1 now i want to build VM-2 on other compute nodes and want to do IP takeover | |