| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-31 | |||
| 14:23:35 | jangutter | sahid: I dunno, this is weird territory for me... literally the only code I could find used iproute2 to enable "multiqueue" and that's it. | |
| 14:23:52 | jaypipes | mriedem: IIRC, that was my original suggestion that I could do that in the online DB migration | |
| 14:24:14 | jaypipes | mriedem: but it was decided that the single CONF option was a simpler solution for the majority case. | |
| 14:24:37 | sahid | jangutter: it's not something that will be used by vrouter so? | |
| 14:24:51 | jaypipes | mriedem: part of the issue with that is that I would have to import nova objects and DB into the placement object files in order to grab instance information. | |
| 14:24:59 | jangutter | sahid: It seems to be a bit of an aberration, almost all of the other VIF types can learn the multiqueue configs from the hypervisor, just TAP seem to need something on the plug side. | |
| 14:25:31 | dansmith | jaypipes: which we can't do post-split, so we shouldn't do now, if we're serious about maintaining that separation | |
| 14:25:32 | jangutter | sahid: yeah I think vrouter's the only remaining bit that requires that... | |
| 14:26:18 | jangutter | sahid: I think things like vhostuser negotiates it over-the-wire or something. I haven't looked closely. | |
| 14:26:24 | jaypipes | dansmith: right, that was the concern. | |
| 14:28:02 | mriedem | jaypipes: sure i get that, | |
| 14:28:11 | mriedem | the heal_allocations CLI is all REST API stuff wrt placement | |
| 14:28:21 | mriedem | so there is no mixed db naughtiness | |
| 14:29:47 | mriedem | if we ever get to counting quotas using placement, i think we'll have to fix those consumer records to use the same project/user as the instance, one way or another | |
| 14:30:07 | mriedem | because counting quotas against a "real" project/user won't include any usage marked against the config options | |
| 14:30:43 | mriedem | heal_allocations could be told to also compare the allocation project/user against the instance project/user and if not the same, update the allocation | |
| 14:30:43 | dansmith | yes, definitely, but I think jaypipes' plan was to make sure that happened first, and I thought heal_allocations was the way | |
| 14:30:45 | jaypipes | mriedem: agreed. | |
| 14:30:53 | jaypipes | dansmith: yes, it was. | |
| 14:31:21 | mriedem | right so i think we're on the same page: fill the hole we have now with the config options, then rely on heal_allocations to properly update the allocations per instance later | |
| 14:31:31 | dansmith | yup | |
| 14:31:32 | mriedem | ok | |
| 14:31:34 | mriedem | whew | |
| 14:31:40 | jaypipes | perfecto | |
| 14:31:47 | mriedem | we should write this down... | |
| 14:33:01 | mriedem | i might amend something into https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/add-consumer-generation.html#upgrade-impact later just as a reminder | |
| 14:33:37 | melwitt | ++ reminders are good | |
| 14:33:54 | sahid | jangutter: ok so i just checked on code, yes it seems that vrouter is the only which needs to created tap device with multiqueue bit so far | |
| 14:34:00 | sahid | so let keep that simple | |
| 14:34:18 | melwitt | takashin: thanks for fixing the meeting times on the agenda -- I didn't realize I had incorrectly swapped them after summit week | |
| 14:34:34 | jangutter | sahid: yeah, it can spiral out of control quickly, txqueues, rxqueues.... | |
| 14:36:56 | sahid | jangutter: yes right... | |
| 14:37:57 | sahid | but i think we do not have any other choice, "normally" it's libvirt which creates the tap and configures the queues | |
| 14:38:38 | sahid | but vrouter is using ethernet interface | |
| 14:42:05 | takashin | melwitt: I'm glad to help. | |
| 14:44:12 | dansmith | mriedem: so I need to raise upper constraints for oslo.messaging prior to my patch to use the new version yeah? that's not automatic I assume... | |
| 14:45:24 | dansmith | actually maybe it gets auto-proposed/ | |
| 14:45:31 | dansmith | I see a bunch of those in requirements right now | |
| 14:48:32 | mriedem | it's automatic | |
| 14:48:42 | mriedem | you'll have to bump lower-constraints in your nova change | |
| 14:49:19 | dansmith | um | |
| 14:49:42 | mriedem | upper-constraints in the requirements repo gets updated automatically via a bot | |
| 14:49:57 | dansmith | okay I guess I'm not sure why we have lower-constraints in our tree | |
| 14:50:01 | mriedem | the in-tree lower-constraints file (and requirements.txt i guess) in nova will need to be updated | |
| 14:50:21 | mriedem | it's in all trees, to run the lower-constraints job | |
| 14:50:26 | mriedem | since all other jobs run against upper-constraints | |
| 14:50:40 | mriedem | and to decouple g-r from a minimum baseline across all projects | |
| 14:50:52 | dansmith | I guess because we how have our own set of lowers or whatev3er | |
| 14:50:53 | dansmith | okay | |
| 14:50:53 | mriedem | i guess, i'm not really following all of that work, but it's what i know | |
| 14:50:58 | mriedem | yup | |
| 14:55:26 | jangutter | sahid: yeah, vrouter is using the 'this taints libvirt' ethernet interface. Maybe with Tungsten Fabric there's a chance to add that to libvirt, but who knows. | |
| 14:55:49 | belmorei_ | mriedem: do you think that I can already use: https://review.openstack.org/#/c/565886 | |
| 15:04:07 | belmoreira | mriedem do you think that I can already use: https://review.openstack.org/#/c/565886 | |
| 15:06:46 | jaypipes | sahid: "The whole NUMA thing is basically a cartography for the scheduler to help it making good decisions." <-- you're referring to the kernel process scheduler, right? not the nova-scheduler,... | |
| 15:07:08 | mriedem | belmoreira: what are you trying to solve with it? | |
| 15:07:14 | mriedem | missing allocations? | |
| 15:07:15 | sahid | jaypipes: yes yes :) | |
| 15:07:23 | jaypipes | sahid: ok, just making sure :) | |
| 15:07:36 | belmoreira | mriedem we had an issue with ironic that triggered the recreation of all compute_nodes; now the new compute_nodes can't be registered in placement because the previous resource_provider/allocations | |
| 15:08:12 | mriedem | so can't create the new resource provider because another with the same name/uuid already exists right? | |
| 15:08:15 | jaypipes | sahid, stephenfin: you'll note I tried as much as possible to remove any and all mention of NUMA topology (host or guest/virtual) in the CPU resources spec. | |
| 15:08:23 | mriedem | and can't delete the existing one because the existing one has allocations already? | |
| 15:08:48 | belmoreira | mriedem: yes | |
| 15:08:54 | belmoreira | the cleanest way to recover would be to delete everything related to ironic resources providers and recreate again | |
| 15:09:06 | jaypipes | sahid, stephenfin: at least in so much as the NUMA topology stuff wasn't really germane to the "request different providers supply some X amount of dedicated or shared CPU resources for the guest" | |
| 15:09:25 | belmoreira | unfortunately I have ~2000 nodes in this state | |
| 15:09:28 | mriedem | belmoreira: yeah i think what i'd probably do is pick one resource provider / allocation combo since they should be 1:1 with ironic, | |
| 15:09:40 | mriedem | clean it up using the osc-placement CLI | |
| 15:09:57 | mriedem | and then run heal_allocations but probably want to put a cap on the --max-count | |
| 15:09:59 | stephenfin | jaypipes: ack. In a meeting/working on NUMA-aware vswitches, but I'll take a look before EOD | |
| 15:10:12 | mriedem | belmoreira: this is why i had a note in the code about allowing passing in a single instance id to heal | |
| 15:10:17 | mriedem | that would be ideal in this case as a test | |
| 15:10:38 | jaypipes | stephenfin: no worries, you already reviewed again (and I will update to your comments) | |
| 15:10:46 | belmoreira | mriedem I will first test all on copies of the DBs | |
| 15:11:04 | mriedem | belmoreira: ok yeah definitely doing it in a test env would be good for both of us :) | |
| 15:12:26 | belmoreira | other use-case for this code is updating ironic allocations that were created before a resource_class was defined | |
| 15:13:38 | mriedem | so the existing allocations for the ironic instance are against vcpu/memory_mb/disk_gb and you want to change then to be against custom_baremetal_belmiro, yes? | |
| 15:13:46 | mriedem | *them | |
| 15:14:31 | belmoreira | mriedem: but the recreation of the compute_node in nova is something that needs to be reviewed; has been creating few issues. Don't know yet why this happened... | |
| 15:14:35 | mriedem | that's probably where the --force option mentioned in the comments would come in, since currently the command will skip any instances that already have allocations, not attempt to overwrite them | |
| 15:15:27 | belmoreira | mriedem:yes, that's way I was pointing this as another use-case | |
| 15:15:27 | mriedem | belmoreira: so i thought that sounded very much like https://review.openstack.org/#/c/508555/ | |
| 15:16:53 | belmoreira | mriedem: this was not rebalance... we had an operator working in ironic/nova-compute and know is very hard to track why this happened | |
| 15:18:07 | belmoreira | mriedem: but anyway it looks a very dangerous operation to do when having placement. Before placement this was OK (recreating a new compute_node) | |
| 15:18:13 | mriedem | was the related nova-compute service deleted by any chance? | |
| 15:18:37 | belmoreira | mriedem: no, I don't yet | |
| 15:18:40 | mriedem | ok | |
| 15:18:53 | mriedem | for ironic the compute node record should have a predictable uuid | |
| 15:19:21 | mriedem | unless... | |
| 15:19:33 | mriedem | we don't use the ironic node uuid when creating the compute node | |
| 15:20:37 | stephenfin | mriedem, dansmith: We've been talking about extending the limits dict but I just noticed this https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1968-L1991 | |
| 15:20:39 | mriedem | looks like we don't https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L580 | |
| 15:21:10 | stephenfin | mriedem, dansmith: Given that we've essentially deprecated 'limits' and have 'requested_networks' available to us, couldn't I just use that as is? | |
| 15:21:14 | jaypipes | stephenfin: who's been talking about extending the limits dict? | |
| 15:21:16 | belmoreira | mriedem: that's the problem. When the compute_node is recreated we get a new uuid | |
| 15:21:18 | mriedem | belmoreira: so i think in that code, resources['hypervisor_hostname'] for ironic is always the node uuid, | |
| 15:21:26 | stephenfin | jaypipes: not me | |
| 15:21:27 | mriedem | we should use that when creating the compute node record if we think it's new | |
| 15:21:35 | mriedem | belmoreira: yup, seems like a trivial fix | |