| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-31 | |||
| 14:30:53 | jaypipes | dansmith: yes, it was. | |
| 14:31:21 | mriedem | right so i think we're on the same page: fill the hole we have now with the config options, then rely on heal_allocations to properly update the allocations per instance later | |
| 14:31:31 | dansmith | yup | |
| 14:31:32 | mriedem | ok | |
| 14:31:34 | mriedem | whew | |
| 14:31:40 | jaypipes | perfecto | |
| 14:31:47 | mriedem | we should write this down... | |
| 14:33:01 | mriedem | i might amend something into https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/add-consumer-generation.html#upgrade-impact later just as a reminder | |
| 14:33:37 | melwitt | ++ reminders are good | |
| 14:33:54 | sahid | jangutter: ok so i just checked on code, yes it seems that vrouter is the only which needs to created tap device with multiqueue bit so far | |
| 14:34:00 | sahid | so let keep that simple | |
| 14:34:18 | melwitt | takashin: thanks for fixing the meeting times on the agenda -- I didn't realize I had incorrectly swapped them after summit week | |
| 14:34:34 | jangutter | sahid: yeah, it can spiral out of control quickly, txqueues, rxqueues.... | |
| 14:36:56 | sahid | jangutter: yes right... | |
| 14:37:57 | sahid | but i think we do not have any other choice, "normally" it's libvirt which creates the tap and configures the queues | |
| 14:38:38 | sahid | but vrouter is using ethernet interface | |
| 14:42:05 | takashin | melwitt: I'm glad to help. | |
| 14:44:12 | dansmith | mriedem: so I need to raise upper constraints for oslo.messaging prior to my patch to use the new version yeah? that's not automatic I assume... | |
| 14:45:24 | dansmith | actually maybe it gets auto-proposed/ | |
| 14:45:31 | dansmith | I see a bunch of those in requirements right now | |
| 14:48:32 | mriedem | it's automatic | |
| 14:48:42 | mriedem | you'll have to bump lower-constraints in your nova change | |
| 14:49:19 | dansmith | um | |
| 14:49:42 | mriedem | upper-constraints in the requirements repo gets updated automatically via a bot | |
| 14:49:57 | dansmith | okay I guess I'm not sure why we have lower-constraints in our tree | |
| 14:50:01 | mriedem | the in-tree lower-constraints file (and requirements.txt i guess) in nova will need to be updated | |
| 14:50:21 | mriedem | it's in all trees, to run the lower-constraints job | |
| 14:50:26 | mriedem | since all other jobs run against upper-constraints | |
| 14:50:40 | mriedem | and to decouple g-r from a minimum baseline across all projects | |
| 14:50:52 | dansmith | I guess because we how have our own set of lowers or whatev3er | |
| 14:50:53 | mriedem | i guess, i'm not really following all of that work, but it's what i know | |
| 14:50:53 | dansmith | okay | |
| 14:50:58 | mriedem | yup | |
| 14:55:26 | jangutter | sahid: yeah, vrouter is using the 'this taints libvirt' ethernet interface. Maybe with Tungsten Fabric there's a chance to add that to libvirt, but who knows. | |
| 14:55:49 | belmorei_ | mriedem: do you think that I can already use: https://review.openstack.org/#/c/565886 | |
| 15:04:07 | belmoreira | mriedem do you think that I can already use: https://review.openstack.org/#/c/565886 | |
| 15:06:46 | jaypipes | sahid: "The whole NUMA thing is basically a cartography for the scheduler to help it making good decisions." <-- you're referring to the kernel process scheduler, right? not the nova-scheduler,... | |
| 15:07:08 | mriedem | belmoreira: what are you trying to solve with it? | |
| 15:07:14 | mriedem | missing allocations? | |
| 15:07:15 | sahid | jaypipes: yes yes :) | |
| 15:07:23 | jaypipes | sahid: ok, just making sure :) | |
| 15:07:36 | belmoreira | mriedem we had an issue with ironic that triggered the recreation of all compute_nodes; now the new compute_nodes can't be registered in placement because the previous resource_provider/allocations | |
| 15:08:12 | mriedem | so can't create the new resource provider because another with the same name/uuid already exists right? | |
| 15:08:15 | jaypipes | sahid, stephenfin: you'll note I tried as much as possible to remove any and all mention of NUMA topology (host or guest/virtual) in the CPU resources spec. | |
| 15:08:23 | mriedem | and can't delete the existing one because the existing one has allocations already? | |
| 15:08:48 | belmoreira | mriedem: yes | |
| 15:08:54 | belmoreira | the cleanest way to recover would be to delete everything related to ironic resources providers and recreate again | |
| 15:09:06 | jaypipes | sahid, stephenfin: at least in so much as the NUMA topology stuff wasn't really germane to the "request different providers supply some X amount of dedicated or shared CPU resources for the guest" | |
| 15:09:25 | belmoreira | unfortunately I have ~2000 nodes in this state | |
| 15:09:28 | mriedem | belmoreira: yeah i think what i'd probably do is pick one resource provider / allocation combo since they should be 1:1 with ironic, | |
| 15:09:40 | mriedem | clean it up using the osc-placement CLI | |
| 15:09:57 | mriedem | and then run heal_allocations but probably want to put a cap on the --max-count | |
| 15:09:59 | stephenfin | jaypipes: ack. In a meeting/working on NUMA-aware vswitches, but I'll take a look before EOD | |
| 15:10:12 | mriedem | belmoreira: this is why i had a note in the code about allowing passing in a single instance id to heal | |
| 15:10:17 | mriedem | that would be ideal in this case as a test | |
| 15:10:38 | jaypipes | stephenfin: no worries, you already reviewed again (and I will update to your comments) | |
| 15:10:46 | belmoreira | mriedem I will first test all on copies of the DBs | |
| 15:11:04 | mriedem | belmoreira: ok yeah definitely doing it in a test env would be good for both of us :) | |
| 15:12:26 | belmoreira | other use-case for this code is updating ironic allocations that were created before a resource_class was defined | |
| 15:13:38 | mriedem | so the existing allocations for the ironic instance are against vcpu/memory_mb/disk_gb and you want to change then to be against custom_baremetal_belmiro, yes? | |
| 15:13:46 | mriedem | *them | |
| 15:14:31 | belmoreira | mriedem: but the recreation of the compute_node in nova is something that needs to be reviewed; has been creating few issues. Don't know yet why this happened... | |
| 15:14:35 | mriedem | that's probably where the --force option mentioned in the comments would come in, since currently the command will skip any instances that already have allocations, not attempt to overwrite them | |
| 15:15:27 | mriedem | belmoreira: so i thought that sounded very much like https://review.openstack.org/#/c/508555/ | |
| 15:15:27 | belmoreira | mriedem:yes, that's way I was pointing this as another use-case | |
| 15:16:53 | belmoreira | mriedem: this was not rebalance... we had an operator working in ironic/nova-compute and know is very hard to track why this happened | |
| 15:18:07 | belmoreira | mriedem: but anyway it looks a very dangerous operation to do when having placement. Before placement this was OK (recreating a new compute_node) | |
| 15:18:13 | mriedem | was the related nova-compute service deleted by any chance? | |
| 15:18:37 | belmoreira | mriedem: no, I don't yet | |
| 15:18:40 | mriedem | ok | |
| 15:18:53 | mriedem | for ironic the compute node record should have a predictable uuid | |
| 15:19:21 | mriedem | unless... | |
| 15:19:33 | mriedem | we don't use the ironic node uuid when creating the compute node | |
| 15:20:37 | stephenfin | mriedem, dansmith: We've been talking about extending the limits dict but I just noticed this https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1968-L1991 | |
| 15:20:39 | mriedem | looks like we don't https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L580 | |
| 15:21:10 | stephenfin | mriedem, dansmith: Given that we've essentially deprecated 'limits' and have 'requested_networks' available to us, couldn't I just use that as is? | |
| 15:21:14 | jaypipes | stephenfin: who's been talking about extending the limits dict? | |
| 15:21:16 | belmoreira | mriedem: that's the problem. When the compute_node is recreated we get a new uuid | |
| 15:21:18 | mriedem | belmoreira: so i think in that code, resources['hypervisor_hostname'] for ironic is always the node uuid, | |
| 15:21:26 | stephenfin | jaypipes: not me | |
| 15:21:27 | mriedem | we should use that when creating the compute node record if we think it's new | |
| 15:21:35 | mriedem | belmoreira: yup, seems like a trivial fix | |
| 15:21:49 | mriedem | belmoreira: not sure if you have enough words to open a bug? | |
| 15:22:39 | mriedem | stephenfin: my guess is requested_networks aren't passed back to the cell conductor during a reschedule | |
| 15:22:55 | mriedem | oh it looks like they are https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1890 | |
| 15:22:56 | belmoreira | This is related with https://review.openstack.org/#/c/508555/ | |
| 15:23:16 | belmoreira | sorry not this one... | |
| 15:23:34 | stephenfin | mriedem: Yeah, I checked - the info is there | |
| 15:24:16 | stephenfin | For migrate and unshelve, I would need to check 'instance.info_cache.network_info' but that's a-ok | |
| 15:25:03 | kashyap | stephenfin: Thanks for the review here, https://review.openstack.org/#/c/567258/. Yep, that could go... | |
| 15:25:32 | kashyap | I'll respin. | |
| 15:26:37 | mriedem | belmoreira: hmm, this should discover if there is already a compute node in the db with the same hypervisor_hostname, which is the ironic node uuid https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L518 | |
| 15:26:44 | belmoreira | mriedem: https://bugs.launchpad.net/nova/+bug/1771806 (the main issue here is the new UUID that's generated. Don't think is required to have a different bug) | |
| 15:26:45 | openstack | Launchpad bug 1771806 in OpenStack Compute (nova) "Ironic nova-compute failover creates new resource provider removing the resource_provider_aggregates link " [Medium,Confirmed] - Assigned to Surya Seetharaman (tssurya) | |
| 15:26:50 | mriedem | so it might not be a rebalance, but that code should have found it | |
| 15:26:50 | openstackgerrit | Merged openstack/nova stable/ocata: Fix shelving a paused instance https://review.openstack.org/569655 | |
| 15:27:29 | mriedem | unless you're hitting https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L531 | |
| 15:27:50 | mriedem | belmoreira: yeah, and in https://bugs.launchpad.net/nova/+bug/1771806 i asked why https://review.openstack.org/#/c/508555/ doesn't fix the issue, and i'm still kind of wondering that | |
| 15:27:51 | openstack | Launchpad bug 1771806 in OpenStack Compute (nova) "Ironic nova-compute failover creates new resource provider removing the resource_provider_aggregates link " [Medium,Confirmed] - Assigned to Surya Seetharaman (tssurya) | |
| 15:28:02 | mriedem | are you seeing the error from this condition? https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L531 | |