| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-15 | |||
| 18:53:41 | cdent | for me it's in part a learned skill to intentionally miss out on some stuff | |
| 18:55:03 | mriedem | could have sworn something about the IronicNodeState object was different such that the filter would fail with it | |
| 18:55:12 | mriedem | also, it's not like we actually have pci requests in these CI jobs | |
| 18:55:30 | edleafe | I tried to learn to skim back in college. I always felt like I never got anything out of it, so I went back to reading details | |
| 18:55:58 | edleafe | efried: excellent recap | |
| 18:55:59 | dansmith | mriedem: that's true, I guess a flavor or request with pci or numa could break if there are ironic hosts in there somehow, | |
| 18:56:09 | dansmith | but I dunno what that would be really | |
| 18:56:29 | efried | edleafe: Thanks. | |
| 18:56:36 | mriedem | dansmith: the other thing might have been something to do with allocation ratios, but grasping at straws | |
| 18:57:29 | dansmith | we don't have ratios for those types though | |
| 18:58:04 | mriedem | goes into the numa topology limits | |
| 18:58:08 | mriedem | the cpu and ram allocation ratois | |
| 18:58:55 | cfriesen | mriedem: we have a private patch to enable ironic and regular nodes...had to make some of the scheduler filters check the hypervisor type. | |
| 18:59:12 | mriedem | cfriesen: is there anything you guys don't have a private patch for? | |
| 18:59:33 | cfriesen | mriedem: we try to upstream stuff, but it takes forever | |
| 18:59:40 | mgoddard_ | mriedem: that's good news! | |
| 19:00:01 | cfriesen | mriedem: plus, we only need to worry about one hypervisor | |
| 19:00:16 | mriedem | mgoddard_: well, it's kind of a false positive i think | |
| 19:00:34 | mriedem | cfriesen: it takes even longer when you don't even propose them | |
| 19:00:45 | dansmith | mriedem: it's not a false positive, it's just not comprehensive.. it means something, it just doesn't mean it all works fine :) | |
| 19:01:39 | mriedem | i think host_topology_and_format_from_host might be the thing | |
| 19:02:20 | mriedem | that's just always None for ironic i think | |
| 19:02:22 | cfriesen | mriedem: yeah, I know. I don't control how much upstreaming time I get. Looks like we modified NUMATopologyFilter to check the hypervisor type, and modified AggregateInstanceExtraSpecsFilter to ignore the "baremetal" and "storage" keys. | |
| 19:02:45 | dansmith | mriedem: if that breaks if it's none, that probably also precludes using any two hypervisors together where one is libvirt with numa and the other doesn't support it, right? | |
| 19:03:14 | mriedem | nvm, that would just filter out that ironic node | |
| 19:03:19 | mriedem | / host | |
| 19:03:38 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: Resize https://review.openstack.org/553583 | |
| 19:03:59 | mriedem | cfriesen: if you could figure out *why* you "Looks like we modified NUMATopologyFilter to check the hypervisor type" that would be nice | |
| 19:04:13 | mriedem | because at some point i thought these wouldn't work but i can't figure out why now | |
| 19:04:24 | dansmith | yeah at least upstreaming the bug would be worthwhile | |
| 19:04:33 | mriedem | ++ | |
| 19:04:42 | dansmith | although they do that, so if you did in this case, then .. cool :) | |
| 19:04:56 | cfriesen | mriedem: let me check with the author | |
| 19:10:48 | sean-k-mooney | mriedem: cfriesen perhaps because you wanted to avoid qemu hosts when there are numa requests? | |
| 19:11:59 | mgoddard_ | perhaps the bug is just that the NUMA filter doesn't work for ironic, rather than that it rejects all bare metal hosts? | |
| 19:12:11 | cfriesen | sean-k-mooney: don't think so, this was specifically part of allowing one nova-scheduler to handle both libvirt/kvm and ironic compute nodes | |
| 19:12:29 | sean-k-mooney | cfriesen: ah ok | |
| 19:13:41 | sean-k-mooney | mgoddard_: well if i ask for a 2 numa node instance the ironic should be able to select a node with 2 numa nodes however i dont think ironic adds numa info into the compute nodes table for the filter to use | |
| 19:13:53 | mgoddard_ | exactly | |
| 19:14:08 | mgoddard_ | same with cpu pinning | |
| 19:14:19 | sean-k-mooney | mgoddard_: no cpu pinning is different | |
| 19:14:20 | mgoddard_ | and hyperthreading | |
| 19:14:35 | cfriesen | mriedem: back to the nova-compute service delete issue, when deleting a service via the API, nova.db.sqlalchemy.api.service_destroy() will soft-delete both the service and the service and the compute_node entry, but placement is still around. Then we create the compute node again and get new service and compute_node entries with a different uuid but the same hostname. | |
| 19:14:36 | sean-k-mooney | cpu pinning does not make sense in a phyical server context | |
| 19:14:39 | dansmith | it wouldn't matter these days anyway, ironic nodes report a CUSTOM_IRONIC_FOO resource and no cpu/mem | |
| 19:15:36 | mgoddard_ | sean-k-mooney: ok, you're right about pinning. hyperthreading could (but doesn't) work though | |
| 19:15:53 | dansmith | cfriesen: mriedem because we store service_id in the compute node, so we won't find the existing one when we re-create the service | |
| 19:16:15 | dansmith | cfriesen: mriedem I bet that api was never updated when we added the node concept.. it probably needs to delete the node(s) as well when it does that | |
| 19:16:19 | sean-k-mooney | mgoddard_: hyperthreading is a bios config option and should work but nova does not allow you to enable hyperthreading as a flavor extra spec | |
| 19:16:21 | dansmith | (and thus placement( | |
| 19:16:49 | sean-k-mooney | mgoddard_: the closest you have to that is setting the tread count in the cpu topology extra specs | |
| 19:17:24 | mriedem | i didn't realize that service_destroy also deleted the related compute node record | |
| 19:17:40 | cfriesen | me either...had to go read the code. | |
| 19:18:55 | dansmith | cfriesen: I thought you said it didn't? | |
| 19:18:56 | mriedem | ok so if service delete also deletes the compute node record, then yeah we should also remove the compute node RP in placement | |
| 19:19:09 | mriedem | i said deleting the service didn't delete the compute node record | |
| 19:19:19 | dansmith | oh okay | |
| 19:19:31 | mriedem | there was some bug about a year ago we were both talking about this | |
| 19:19:41 | mriedem | but https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L404 | |
| 19:21:31 | cfriesen | I should be able to open a bug and post a WIP fix later today | |
| 19:21:31 | mriedem | https://github.com/openstack/nova/commit/f0d44c5b09f3f3c84038d40b621bb629a1f8110e#diff-3104166b3e802b86db6c5fa92ad08f43 | |
| 19:21:36 | mriedem | ^ is why i thought this | |
| 19:23:28 | mriedem | see the exchange between myself and alex | |
| 19:24:23 | mriedem | so in this case, they deleted the service (and compute node record) but didn't stop the nova-compute service, | |
| 19:24:32 | mriedem | so it re-created the compute node record | |
| 19:25:27 | mriedem | so in that bug, when they listed compute nodes, the api tries to find the related service which was deleted and the api blows up | |
| 19:25:34 | mriedem | because you don't recreate the service until you restart the service | |
| 19:26:16 | melwitt | gibi: I'm working on the nova/neutron session summary, would be helpful if you could fill in any gaps for the bandwidth-based scheduling agreements/decisions when you get a chance https://etherpad.openstack.org/p/nova-ptg-rocky-neutron-summary | |
| 19:26:25 | mriedem | melwitt: he's on vacation | |
| 19:27:01 | melwitt | urgh right | |
| 19:27:34 | melwitt | got focused only on the april vacation dates | |
| 19:27:37 | mriedem | cfriesen: yeah so to summarize that, deleting the service deletes the compute node record, and the compute node record shouldn't show back up unless the service is running | |
| 19:28:04 | dansmith | ...and we should delete the RP in placement | |
| 19:28:09 | mriedem | what about the host mapping? | |
| 19:28:20 | mriedem | that's also orphaned right? | |
| 19:28:20 | dansmith | we probably should delete it in placement before we delete in our db even | |
| 19:28:26 | dansmith | yeah | |
| 19:28:59 | mriedem | so, | |
| 19:29:13 | mriedem | 1. delete RP in placement (because if we fail to delete the compute node or the service, it will get re-created), | |
| 19:29:16 | mriedem | 2. delete the service/node, | |
| 19:29:20 | mriedem | 3. delete the host mapping | |
| 19:29:21 | mriedem | ? | |
| 19:29:39 | dansmith | well, 1. also because if we fail, they can't do it again if we delete the local entry | |
| 19:30:00 | dansmith | technically, we can find the service record again if we delete the host mapping, | |
| 19:30:13 | dansmith | so probably good to delete that first, then node, then service last of all | |
| 19:30:20 | cfriesen | what's the host mapping used for? | |
| 19:30:21 | dansmith | so placement, mapping, node, service | |
| 19:30:26 | dansmith | cfriesen: finding it by name in a cell | |
| 19:30:27 | mriedem | cfriesen: to map the host | |
| 19:30:29 | cfriesen | ah, right | |
| 19:30:49 | mriedem | node and service are in the same db api call, | |
| 19:30:57 | mriedem | unless we delete the compute node record(s?) directly in the api | |
| 19:31:15 | mriedem | which would be my preference b/c the auto/implicit delete in the db api is kind of gross | |
| 19:32:13 | dansmith | same db call, but node could go first | |
| 19:32:22 | dansmith | but yeah, don't so much care about those two, | |
| 19:32:51 | dansmith | as I'm not sure there are any FKs that would prevent us from deleting one and not the other | |
| 19:33:46 | mriedem | compute_nodes.service_id doesn't have a FK on it | |
| 19:33:55 | dansmith | yeah | |
| 19:34:08 | cfriesen | dansmith: what are the implications of a stale host mapping? | |
| 19:34:18 | dansmith | cfriesen: cruft | |