| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-31 | |||
| 14:11:09 | mriedem | jaypipes: yeah i forgot about that | |
| 14:11:17 | dansmith | right, but it muddies the water for a split | |
| 14:12:16 | mriedem | yeah if we have an online data migration i'm not sure we need an upgrade status check thing | |
| 14:13:34 | mriedem | i think i was thinking, | |
| 14:13:53 | mriedem | somehow healing the allocations / creating missing consumers based on the actual instance for the allocation, rather than the config options | |
| 14:13:59 | mriedem | which is what jay's online data migration is using it looks like | |
| 14:14:51 | dansmith | hmm | |
| 14:15:17 | dansmith | I mean, I guess we can do that now, but we won't be able to do things like that past the split, so I would think we should stick to being cleanly separated even before | |
| 14:15:42 | jaypipes | mriedem: your heal_allocations() work would already do that (once the always create consumer patch is merged, of course) | |
| 14:16:00 | mriedem | jaypipes: the heal_allocations CLI won't do anything if the instance already has allocations | |
| 14:16:06 | mriedem | but it doesn't check if the project/user is missing | |
| 14:16:17 | jaypipes | mriedem: ah, yes, true nuf | |
| 14:16:41 | mriedem | https://review.openstack.org/#/c/565886/9/nova/cmd/manage.py@1794 | |
| 14:21:14 | sahid | jangutter: i have question for you in the osvif patch | |
| 14:21:27 | jangutter | shoot! | |
| 14:21:50 | mriedem | jaypipes: having said that, we could make heal_allocations look to see if the existing allocations are missing consumer/project and re-post the allocations w/ the consumer/project if so | |
| 14:22:40 | sahid | jangutter: is that could help whether we have that new virtio_multiqueu attribute an integer, so we can pass the exact number of queue enabled? | |
| 14:23:35 | jangutter | sahid: I dunno, this is weird territory for me... literally the only code I could find used iproute2 to enable "multiqueue" and that's it. | |
| 14:23:52 | jaypipes | mriedem: IIRC, that was my original suggestion that I could do that in the online DB migration | |
| 14:24:14 | jaypipes | mriedem: but it was decided that the single CONF option was a simpler solution for the majority case. | |
| 14:24:37 | sahid | jangutter: it's not something that will be used by vrouter so? | |
| 14:24:51 | jaypipes | mriedem: part of the issue with that is that I would have to import nova objects and DB into the placement object files in order to grab instance information. | |
| 14:24:59 | jangutter | sahid: It seems to be a bit of an aberration, almost all of the other VIF types can learn the multiqueue configs from the hypervisor, just TAP seem to need something on the plug side. | |
| 14:25:31 | dansmith | jaypipes: which we can't do post-split, so we shouldn't do now, if we're serious about maintaining that separation | |
| 14:25:32 | jangutter | sahid: yeah I think vrouter's the only remaining bit that requires that... | |
| 14:26:18 | jangutter | sahid: I think things like vhostuser negotiates it over-the-wire or something. I haven't looked closely. | |
| 14:26:24 | jaypipes | dansmith: right, that was the concern. | |
| 14:28:02 | mriedem | jaypipes: sure i get that, | |
| 14:28:11 | mriedem | the heal_allocations CLI is all REST API stuff wrt placement | |
| 14:28:21 | mriedem | so there is no mixed db naughtiness | |
| 14:29:47 | mriedem | if we ever get to counting quotas using placement, i think we'll have to fix those consumer records to use the same project/user as the instance, one way or another | |
| 14:30:07 | mriedem | because counting quotas against a "real" project/user won't include any usage marked against the config options | |
| 14:30:43 | mriedem | heal_allocations could be told to also compare the allocation project/user against the instance project/user and if not the same, update the allocation | |
| 14:30:43 | dansmith | yes, definitely, but I think jaypipes' plan was to make sure that happened first, and I thought heal_allocations was the way | |
| 14:30:45 | jaypipes | mriedem: agreed. | |
| 14:30:53 | jaypipes | dansmith: yes, it was. | |
| 14:31:21 | mriedem | right so i think we're on the same page: fill the hole we have now with the config options, then rely on heal_allocations to properly update the allocations per instance later | |
| 14:31:31 | dansmith | yup | |
| 14:31:32 | mriedem | ok | |
| 14:31:34 | mriedem | whew | |
| 14:31:40 | jaypipes | perfecto | |
| 14:31:47 | mriedem | we should write this down... | |
| 14:33:01 | mriedem | i might amend something into https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/add-consumer-generation.html#upgrade-impact later just as a reminder | |
| 14:33:37 | melwitt | ++ reminders are good | |
| 14:33:54 | sahid | jangutter: ok so i just checked on code, yes it seems that vrouter is the only which needs to created tap device with multiqueue bit so far | |
| 14:34:00 | sahid | so let keep that simple | |
| 14:34:18 | melwitt | takashin: thanks for fixing the meeting times on the agenda -- I didn't realize I had incorrectly swapped them after summit week | |
| 14:34:34 | jangutter | sahid: yeah, it can spiral out of control quickly, txqueues, rxqueues.... | |
| 14:36:56 | sahid | jangutter: yes right... | |
| 14:37:57 | sahid | but i think we do not have any other choice, "normally" it's libvirt which creates the tap and configures the queues | |
| 14:38:38 | sahid | but vrouter is using ethernet interface | |
| 14:42:05 | takashin | melwitt: I'm glad to help. | |
| 14:44:12 | dansmith | mriedem: so I need to raise upper constraints for oslo.messaging prior to my patch to use the new version yeah? that's not automatic I assume... | |
| 14:45:24 | dansmith | actually maybe it gets auto-proposed/ | |
| 14:45:31 | dansmith | I see a bunch of those in requirements right now | |
| 14:48:32 | mriedem | it's automatic | |
| 14:48:42 | mriedem | you'll have to bump lower-constraints in your nova change | |
| 14:49:19 | dansmith | um | |
| 14:49:42 | mriedem | upper-constraints in the requirements repo gets updated automatically via a bot | |
| 14:49:57 | dansmith | okay I guess I'm not sure why we have lower-constraints in our tree | |
| 14:50:01 | mriedem | the in-tree lower-constraints file (and requirements.txt i guess) in nova will need to be updated | |
| 14:50:21 | mriedem | it's in all trees, to run the lower-constraints job | |
| 14:50:26 | mriedem | since all other jobs run against upper-constraints | |
| 14:50:40 | mriedem | and to decouple g-r from a minimum baseline across all projects | |
| 14:50:52 | dansmith | I guess because we how have our own set of lowers or whatev3er | |
| 14:50:53 | dansmith | okay | |
| 14:50:53 | mriedem | i guess, i'm not really following all of that work, but it's what i know | |
| 14:50:58 | mriedem | yup | |
| 14:55:26 | jangutter | sahid: yeah, vrouter is using the 'this taints libvirt' ethernet interface. Maybe with Tungsten Fabric there's a chance to add that to libvirt, but who knows. | |
| 14:55:49 | belmorei_ | mriedem: do you think that I can already use: https://review.openstack.org/#/c/565886 | |
| 15:04:07 | belmoreira | mriedem do you think that I can already use: https://review.openstack.org/#/c/565886 | |
| 15:06:46 | jaypipes | sahid: "The whole NUMA thing is basically a cartography for the scheduler to help it making good decisions." <-- you're referring to the kernel process scheduler, right? not the nova-scheduler,... | |
| 15:07:08 | mriedem | belmoreira: what are you trying to solve with it? | |
| 15:07:14 | mriedem | missing allocations? | |
| 15:07:15 | sahid | jaypipes: yes yes :) | |
| 15:07:23 | jaypipes | sahid: ok, just making sure :) | |
| 15:07:36 | belmoreira | mriedem we had an issue with ironic that triggered the recreation of all compute_nodes; now the new compute_nodes can't be registered in placement because the previous resource_provider/allocations | |
| 15:08:12 | mriedem | so can't create the new resource provider because another with the same name/uuid already exists right? | |
| 15:08:15 | jaypipes | sahid, stephenfin: you'll note I tried as much as possible to remove any and all mention of NUMA topology (host or guest/virtual) in the CPU resources spec. | |
| 15:08:23 | mriedem | and can't delete the existing one because the existing one has allocations already? | |
| 15:08:48 | belmoreira | mriedem: yes | |
| 15:08:54 | belmoreira | the cleanest way to recover would be to delete everything related to ironic resources providers and recreate again | |
| 15:09:06 | jaypipes | sahid, stephenfin: at least in so much as the NUMA topology stuff wasn't really germane to the "request different providers supply some X amount of dedicated or shared CPU resources for the guest" | |
| 15:09:25 | belmoreira | unfortunately I have ~2000 nodes in this state | |
| 15:09:28 | mriedem | belmoreira: yeah i think what i'd probably do is pick one resource provider / allocation combo since they should be 1:1 with ironic, | |
| 15:09:40 | mriedem | clean it up using the osc-placement CLI | |
| 15:09:57 | mriedem | and then run heal_allocations but probably want to put a cap on the --max-count | |
| 15:09:59 | stephenfin | jaypipes: ack. In a meeting/working on NUMA-aware vswitches, but I'll take a look before EOD | |
| 15:10:12 | mriedem | belmoreira: this is why i had a note in the code about allowing passing in a single instance id to heal | |
| 15:10:17 | mriedem | that would be ideal in this case as a test | |
| 15:10:38 | jaypipes | stephenfin: no worries, you already reviewed again (and I will update to your comments) | |
| 15:10:46 | belmoreira | mriedem I will first test all on copies of the DBs | |
| 15:11:04 | mriedem | belmoreira: ok yeah definitely doing it in a test env would be good for both of us :) | |
| 15:12:26 | belmoreira | other use-case for this code is updating ironic allocations that were created before a resource_class was defined | |
| 15:13:38 | mriedem | so the existing allocations for the ironic instance are against vcpu/memory_mb/disk_gb and you want to change then to be against custom_baremetal_belmiro, yes? | |
| 15:13:46 | mriedem | *them | |
| 15:14:31 | belmoreira | mriedem: but the recreation of the compute_node in nova is something that needs to be reviewed; has been creating few issues. Don't know yet why this happened... | |
| 15:14:35 | mriedem | that's probably where the --force option mentioned in the comments would come in, since currently the command will skip any instances that already have allocations, not attempt to overwrite them | |
| 15:15:27 | belmoreira | mriedem:yes, that's way I was pointing this as another use-case | |
| 15:15:27 | mriedem | belmoreira: so i thought that sounded very much like https://review.openstack.org/#/c/508555/ | |