| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-19 | |||
| 18:43:47 | mriedem | fixing that would mean needing to check the compute service status for every instance action... | |
| 18:44:19 | sean-k-mooney | awaugama: both cases are "wronge" depending on who you ask hench the sepc for initall allcoation ratios https://review.openstack.org/#/c/552105/ | |
| 18:44:56 | mriedem | on every periodic, we update the resource class inventory allocation ratio based on config that we send to placement https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/compute/resource_tracker.py#L108 | |
| 18:45:09 | mriedem | https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/compute/resource_tracker.py#L952 | |
| 18:45:24 | mriedem | the question is if prov_tree.update_inventory(nodename, inv_data) has a bug thinking that nothing changed | |
| 18:45:25 | mriedem | b/c it's a cache | |
| 18:45:34 | mriedem | and the reportclient itself has a cache of the provider tree | |
| 18:46:04 | mriedem | https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1501 | |
| 18:46:08 | mriedem | "The specified ProviderTree is compared against the local cache. Any changes are flushed back to the placement service. " | |
| 18:47:37 | awaugama | makes sense | |
| 18:48:23 | mriedem | the bug is probably here https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1575-L1576 | |
| 18:48:40 | mriedem | if we have a single compute node resource provider and that doesn't change, both of those sets will be empty | |
| 18:48:43 | cfriesen | mriedem: as I said in the bug, that won't actually fix it, just make the race window smaller. | |
| 18:48:47 | mriedem | and the for loops below won't flush any changes to placement | |
| 18:48:54 | mriedem | cfriesen: isure | |
| 18:48:56 | mriedem | *sure | |
| 18:48:59 | mriedem | b/c of the service group api | |
| 18:49:03 | mriedem | unless you force down the service | |
| 18:49:47 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/compute/resource_tracker.py#L883 | |
| 18:49:47 | mriedem | awaugama: in queens we weren't using that provider tree stuff | |
| 18:50:11 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1112 | |
| 18:50:33 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L850 | |
| 18:50:34 | cdent | because it is friday and I don't feel like filtering, can I just say: god I hate caches, why do we do caches? | |
| 18:50:44 | mriedem | idk | |
| 18:50:53 | mriedem | to avoid calling the placement API 500 times per periodic? | |
| 18:51:13 | mriedem | i just know caches are very tricky | |
| 18:51:15 | mriedem | brittle | |
| 18:51:19 | sean-k-mooney | cdent: so the processor has another way to mess with your view of a sequtially constent exectuion of your program | |
| 18:51:41 | awaugama | mriedem: so basically it's a stale value and placement is never refreshing to pick up the new conf setting? | |
| 18:52:01 | awaugama | or the rp isn't refreshing? | |
| 18:52:31 | sean-k-mooney | cdent: hehe since you are here can i get your input on https://review.openstack.org/#/c/610034/ | |
| 18:52:35 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/compute/provider_tree.py#L124 | |
| 18:52:51 | sean-k-mooney | cdent: actully perhaps the placement channel would be better. | |
| 18:53:00 | mriedem | awaugama: i think the scheduler report client in master/rocky is thinking nothing is changing | |
| 18:53:04 | cdent | yeah, join me over there because I think that may be fixed | |
| 18:53:22 | mriedem | i just had doritos, you probably don't want to kiss me | |
| 18:53:25 | mriedem | at least not open mouth | |
| 18:53:57 | awaugama | this channel gets weird on Fridays | |
| 18:54:25 | mriedem | awaugama: i think on master/rocky the problem is we're not getting here https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1646 | |
| 18:54:41 | mriedem | awaugama: it should be pretty easy to recreate this | |
| 18:55:02 | awaugama | mriedem: I think finucannot was able to reproduce on his system | |
| 18:55:13 | awaugama | think he was just using devstack | |
| 18:55:45 | mriedem | that or https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1112 is short circuirting | |
| 18:55:47 | mriedem | *circuiting | |
| 18:56:06 | mriedem | at one point i had a debug patch for a bunch of this b/c it's really hard to know wtf is going on without any logs | |
| 18:56:51 | awaugama | mriedem, I had to redeploy my system for another feature test, I can see about reproducing next week | |
| 18:56:54 | awaugama | with logs | |
| 18:56:54 | mriedem | i'll see if i can dredge that up | |
| 18:57:55 | mriedem | awaugama: https://review.openstack.org/#/c/597560/ | |
| 18:59:23 | awaugama | cool, I'll make a note of that patch and see if I can get it applied | |
| 18:59:36 | mriedem | the stuff in here is probably still useful https://review.openstack.org/#/c/597560/6/nova/scheduler/client/report.py | |
| 18:59:43 | mriedem | the rest was for debugging a specific thing that is now fixed | |
| 18:59:53 | mriedem | maybe i'll restore and rev that to clean it up | |
| 19:00:05 | mriedem | on top of https://review.openstack.org/#/c/597553/ | |
| 19:19:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log the operation when updating generation in ProviderTree https://review.openstack.org/597553 | |
| 19:19:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add debug logs for when provider inventory changes https://review.openstack.org/597560 | |
| 19:29:28 | mriedem | melwitt: https://bugs.launchpad.net/nova/+bug/1798787 | |
| 19:29:29 | openstack | Launchpad bug 1798787 in OpenStack Compute (nova) "Installation help documentation is incorrect - verify & nova-consoleauth" [Medium,Triaged] | |
| 19:30:26 | mriedem | the install guide tells you to verify nova-consoleauth is running but we don't tell you to install/start it | |
| 19:30:30 | mriedem | b/c that part was removed in rocky | |
| 19:31:05 | melwitt | gah | |
| 19:32:14 | cdent | Is there a bug that is associated with provider tree cache problems discussed above? | |
| 19:32:23 | mriedem | cdent: not that i'm aware of | |
| 19:32:31 | mriedem | i think awaugama hit it in QE | |
| 19:33:02 | awaugama | yeah verifying vcpu weighter feature | |
| 19:33:22 | cdent | awaugama: are you making a bug? If so I want to be sure to follow along | |
| 19:33:57 | awaugama | cdent, I will next week. I need to reinstall my system (did another feature test in the meantime) so I'll need to recollect logs | |
| 19:34:08 | mriedem | is this a tripleo system that takes 3 days? | |
| 19:34:17 | SteelyDan | heh | |
| 19:34:26 | cdent | great, thanks awaugama | |
| 19:34:55 | mriedem | btw, speaking of tripleo | |
| 19:34:55 | awaugama | mriedem: I can probably get it repro'd by EOD Tuesday | |
| 19:34:57 | awaugama | but yeah | |
| 19:35:24 | mriedem | who from the red hat nova cabal can add nova-status upgrade check to tripleo? | |
| 19:35:36 | mriedem | owalsh: ^? | |
| 19:35:42 | SteelyDan | or mschuppert | |
| 19:36:04 | awaugama | yeah one of those two is the nova deployment guy | |
| 19:36:52 | awaugama | their specialty is based on the need | |
| 19:38:08 | openstackgerrit | Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 19:38:25 | sean-k-mooney | awaugama: and the fact that everyone else avoid using tripleo to deploy our dev envs if we can | |
| 19:41:13 | awaugama | fair enough | |
| 19:41:35 | mriedem | SteelyDan: oooo guess what just rotated in https://www.youtube.com/watch?v=jJ9Xk-VoGqo | |
| 19:41:41 | SteelyDan | nice | |
| 19:42:16 | SteelyDan | I enjoyed when this rotated in for me this morning: https://www.youtube.com/watch?v=KCdKBHdPz30 | |
| 19:43:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't persist RequestSpec.requested_destination https://review.openstack.org/611939 | |
| 19:43:20 | openstack | bug 1797580 in OpenStack Compute (nova) "NoValidHost during live migration after cold migrating to a specified host" [High,In progress] https://launchpad.net/bugs/1797580 - Assigned to Matt Riedemann (mriedem) | |
| 19:43:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add regression test for bug 1797580 https://review.openstack.org/611938 | |
| 19:44:08 | mriedem | i know i'd never do it without the fez on | |
| 19:44:14 | openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | |
| 19:44:14 | openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | |
| 19:44:15 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 19:44:15 | openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | |
| 19:44:16 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 19:44:18 | SteelyDan | heck no | |
| 19:44:50 | melwitt | never heard of either of those. I only know the major steely dan hits | |
| 19:45:13 | mriedem | kid charlemagne is a major hit | |
| 19:45:17 | mriedem | all the major dudes know that | |
| 19:45:18 | SteelyDan | well, | |
| 19:45:28 | SteelyDan | she probably means like reelin' in the years and rikki | |
| 19:45:29 | melwitt | major dudes?? | |