| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-20 | |||
| 20:55:47 | efried | But it's all provided by the same NUMA node? | |
| 20:55:47 | cfriesen | efried: if there is no instance numa topology, then nova doesn't specify any affinity. this means the qemu threads are free to float across the whole compute node. when they do a memory allocation they will by default be allocated memory from the numa node they're currently running on. | |
| 20:55:52 | cfriesen | efried: no | |
| 20:55:56 | sean-k-mooney | efried: infact today we do not have a facility to enforce that it comes form the same numa node in nova | |
| 20:56:25 | cfriesen | we explicitly let the host decide....we did for a while restrict it to a single numa node, then removed that for increased density | |
| 20:56:32 | cfriesen | (we being nova) | |
| 20:56:44 | efried | And just so I'm clear, you want that kind of VM to be able to run on the same node as the strictly-affinitized one? | |
| 20:56:52 | efried | s/node/host/ | |
| 20:57:00 | sean-k-mooney | cfriesen: did that ever land upstream. i dont think we ever did a release with that behavior for 4k pages | |
| 20:57:28 | sean-k-mooney | efried: yes i dont see why not | |
| 20:57:42 | efried | Well, y'all seem to be splainin why not. | |
| 20:57:46 | sean-k-mooney | efried: the only way to request numa affined memory in openstack today is via hugepages | |
| 20:57:51 | cfriesen | efried: same compute node, yes. there's a spec under review right now to support shared and dedicated vcpus on the same compute node, and we already support 4KB and 2MB page backing on the same compute node | |
| 20:58:25 | cfriesen | it's just that right now the resource tracking is kind of messed up for 4KB pages | |
| 20:59:01 | sean-k-mooney | cfriesen: yes 4k pages are not tracked in the numa topology blob so we cant numa afine them | |
| 20:59:38 | sean-k-mooney | cfriesen: you can request 4k pages specifically but the is special case code the skips the numa suff for them | |
| 20:59:58 | cfriesen | sean-k-mooney: aren't they part of mempages? | |
| 21:00:09 | openstackgerrit | Merged openstack/nova master: Always pass 'NUMACell.siblings' to _pack_instance_onto_cores' https://review.openstack.org/537364 | |
| 21:00:28 | sean-k-mooney | mempages? | |
| 21:00:38 | cfriesen | NUMACell.mempages | |
| 21:01:04 | sean-k-mooney | you can set hw:mem_page_size=4k but i dont think they are stored in NUMACell.mempages | |
| 21:01:13 | sean-k-mooney | i guess i can check the db one seck | |
| 21:04:22 | sean-k-mooney | that is what the numatopology blob looks like http://paste.openstack.org/show/706682/ | |
| 21:05:05 | sean-k-mooney | cfriesen: so yes "nova_object.data": {"used": 0, "total": 2043576, "reserved": 0, "size_kb": 4} they are there | |
| 21:06:22 | sean-k-mooney | cfriesen: they are actully tracked per numa node but we cant tie teh per numa node values back to the host memory_mb value simply today | |
| 21:06:55 | cfriesen | I think we can, for ones that are strictly pinned. you just subtract the same value from both | |
| 21:07:13 | cfriesen | but for floating ones we don't know the actual per-host-numa-node consumption | |
| 21:07:19 | sean-k-mooney | i mean technically host memory_mb is the sum of all the cells 4k pages but im not sure if they will always agreee | |
| 21:08:49 | sean-k-mooney | cfriesen: the reall issue is the host reserved memory option. that is host wide and we jsut subtract it form the memory_mb value in the code but not sure how to translate that to per numa reserved values | |
| 21:09:10 | mriedem | efried: melwitt: i reckon an upgrade release note will be in order for this endpoint not found thing with the legacy notification payload image_ref_url being an image id rather than a url | |
| 21:09:19 | sean-k-mooney | cfriesen: that said we really need to deprecate it and replace it with a per numa version at some point | |
| 21:09:24 | cfriesen | sean-k-mooney: agreed | |
| 21:09:53 | efried | mriedem: Yeah, sounds like a plan. Though I expect it to be a while before folks quit using api_servers IRL. | |
| 21:10:03 | sean-k-mooney | anyway its 9 so im going to go home and have dinner o/ | |
| 21:10:08 | cfriesen | later | |
| 21:10:11 | efried | ...which (I think) makes the issue moot. | |
| 21:10:11 | mriedem | efried: same | |
| 21:10:26 | efried | This bug came out of a devstack? | |
| 21:10:38 | efried | Cause that's the one place I know for sure we got rid of api_servers. | |
| 21:10:44 | mriedem | yes it did | |
| 21:10:55 | mriedem | "This on devstack with commit id: 5d2add74534719c5670b29152964a60e8f23b42b" | |
| 21:11:22 | melwitt | yeah, release note is always helpful. I do wonder if the field is nullable and if sending nothing would be better than breaking the contract, but meh, not sure | |
| 21:11:26 | efried | But this is good - flushing out these bugs/corner cases in devstack before they hit the proverbial fan in production. | |
| 21:11:59 | mriedem | melwitt: nullable is also breaking the contract a bit | |
| 21:12:13 | mriedem | fwiw, this is already dumb if you're using bfv, | |
| 21:12:21 | mriedem | because instance.image_ref is '' for bfv | |
| 21:12:32 | mriedem | so we're sending an image ref url with no image id in it | |
| 21:12:42 | melwitt | heh, okay | |
| 21:12:46 | cfriesen | sean-k-mooney: prior to 1231c469d (circa 2014) we did pin instances without a numa_topology to a single host numa node | |
| 21:13:32 | melwitt | efried: yeah, we need to check when this broke so we know how far back to backport. and also if it's in ocata then people are probably hitting this in production | |
| 21:14:21 | efried | melwitt: https://review.openstack.org/#/c/490057/ | |
| 21:14:23 | melwitt | it's maybe not "as noticeable" since it's likely only this sync power states periodic that results in notifications maybe | |
| 21:14:54 | melwitt | efried: okay, cool. | |
| 21:15:21 | melwitt | so only need to backport to queens | |
| 21:15:27 | efried | melwitt: And here's the change that removed api_servers from devstack: https://review.openstack.org/#/c/490031/ | |
| 21:16:12 | efried | melwitt: Well, I'll be a little surprised if that's the only place we're using a tokenless auth context to look up a glance endpoint. | |
| 21:16:20 | efried | given the scope of 490057 | |
| 21:19:43 | melwitt | efried: yeah ... probably. but since it's so recent, I think you're right this is being caught a lot sooner than most people upgrade to queens. I was worried it was going to go back to ocata or something crazy. I know you linked the patch earlier but I didn't notice it was pretty recent | |
| 21:22:14 | melwitt | mriedem: what I was thinking with the nullable thing, would be in the worst case if someone had automation parsing image_ref_url and we make it not a url, but empty string isn't one either. not sure if they're equally bad. just thinking out loud | |
| 21:22:40 | mriedem | if someone blindly takes the image_ref_url and makes a GET curl request with it, it's going to blow up for all volume-backed instances | |
| 21:23:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle EndpointNotFound when building image_ref_url in notifications https://review.openstack.org/554703 | |
| 21:23:14 | mriedem | i think in the long ago, you could also create servers by passing an image URL to the imageRef parameter | |
| 21:23:15 | melwitt | yeah. I was thinking before that as far as the format of it. probably overthinking it | |
| 21:23:16 | mriedem | and nova would parse it | |
| 21:29:15 | mriedem | multiattach is fixed with the queens UCA | |
| 21:29:19 | mriedem | https://review.openstack.org/#/c/554667/ | |
| 21:29:30 | mriedem | that's the only thing blocking us from using the queens UCA in devstack now | |
| 21:30:08 | imacdonn | ooh | |
| 21:31:01 | imacdonn | is there any actually need to use the UCA? ZFSSA CI seems to be doing OK with out .. but that's cinder | |
| 21:31:32 | mriedem | well, a few things, | |
| 21:31:37 | mriedem | we get newer libvirt and qemu, | |
| 21:31:51 | mriedem | testing rocky against pike UCA seems weird, we should at least use queens if it's there | |
| 21:32:13 | mriedem | and it allows us to remove these weird workarounds in devstack for the multiattach job to *not* use the UCA b/c the pike UCA didn't have the right package versions, but queens UCA does https://review.openstack.org/#/c/554317/ | |
| 21:32:19 | mriedem | https://review.openstack.org/#/c/554314/ | |
| 21:32:39 | imacdonn | yeah, I have one of those workarounds (per your recommendation) | |
| 21:32:51 | imacdonn | I suppose it does make sense to test nova stuff with the latest virt stuff | |
| 21:33:09 | mriedem | s/nova/openstack/ | |
| 21:33:36 | imacdonn | seems it'd matter more for nova... but yeah | |
| 21:33:54 | mriedem | sure, for libvirt and qemu yes, but you also get newer things like rados and tgtd | |
| 21:33:57 | mriedem | ovs | |
| 21:33:58 | mriedem | etc | |
| 21:35:34 | imacdonn | OK. Has devstack master already been updated to use the Queens UCA? | |
| 21:35:45 | mriedem | https://review.openstack.org/#/c/554314/ | |
| 21:35:52 | mriedem | depends on fixing this bug in nova first | |
| 21:36:06 | imacdonn | k... I'll subscribe to the bug | |
| 21:36:20 | mriedem | get stvnoyes on the oracle phone | |
| 21:37:07 | imacdonn | not sure if he's still in EU, but even if not, he's probably gone for the day (east coast) | |
| 21:37:17 | melwitt | mriedem: +2 | |
| 21:37:51 | mriedem | thanks | |
| 21:39:08 | openstackgerrit | Merged openstack/nova stable/queens: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553817 | |
| 21:39:21 | imacdonn | Something completely unrelated to bounce off you, mriedem (or anyone else) | |
| 21:40:11 | imacdonn | per Queens release notes, I tried to remove neutron.url from my nova.conf, but it seems to be unable to get the neutron endpoint from the service catalog | |
| 21:40:28 | mriedem | efried: replied to all comments in https://review.openstack.org/#/c/554703/ and i don't think any of them are worth changing | |
| 21:40:31 | imacdonn | I had someone in #openstack buddy-check my config, and it seems sane | |
| 21:40:45 | mriedem | imacdonn: there is a bug fix that you need, | |
| 21:40:46 | mriedem | sec | |
| 21:40:55 | imacdonn | k ;) | |
| 21:40:59 | mriedem | imacdonn: https://github.com/openstack/nova/commit/3a3b0f09db318faf1a1ea711a73bb365cab8b233 | |
| 21:41:56 | imacdonn | mriedem: Interesting, Looks pertinent. Will try it. Thanks! | |
| 21:44:32 | efried | mriedem: Brain fart, sorry 'bout that. +1. | |