| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-18 | |||
| 15:22:38 | gibi | jamespage: above sees memory overallocation issues | |
| 15:23:07 | gibi | sean-k-mooney: so you say that if hw:mem_page_size=small is added then we allow mixing small and large instances? | |
| 15:23:10 | sean-k-mooney | you must not under any cirucmstance mix numa and non numa guest or it will break all our memory tracking | |
| 15:23:18 | sean-k-mooney | yes | |
| 15:23:32 | sean-k-mooney | if you use hw:mem_page_size=small its fine | |
| 15:23:35 | sean-k-mooney | brb | |
| 15:25:09 | sean-k-mooney | tl;dr is using hw:mem_page_small will numa affine the guest and track the memory correctly | |
| 15:25:19 | sean-k-mooney | if you dont it will float and cause random OOM events | |
| 15:26:03 | gibi | thanks! | |
| 15:26:04 | sean-k-mooney | placement is not enough to save us since the OOM reaper runs per numa node and also hugepage guest dont supprot oversubsricption | |
| 15:26:26 | sean-k-mooney | so if if you have memeroy over subscrion it wont really work right | |
| 15:27:22 | sean-k-mooney | we have something about this in our downstream docs but basically we say use hostaggreates to prevnt mixing numa and non numa guests | |
| 15:29:16 | sean-k-mooney | jamespage: if you have enuch swap by the way those other instace coudl technicaly boot but the system would likely be unstable so its better to avoid that | |
| 15:29:20 | jamespage | sorry - power outage just at the wrong moment | |
| 15:30:33 | gibi | jamespage: https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html here are the IRC logs if you are dropped | |
| 15:31:31 | sean-k-mooney | jamespage: the reason it can be schduled to the host is in placemnt we report total ram and then only reserver what set in host_reserver_ram or what ever that option is called | |
| 15:31:59 | sean-k-mooney | jamespage: we dont currently automaticlly reserve the hugepage memrory because we dont currenlty track that as a seperate pool | |
| 15:32:25 | sean-k-mooney | so the schduelr is only looking at total-reserved not total -reserved - hugepages | |
| 15:33:23 | sean-k-mooney | actully if its a recent release we dont even have a ram filter anymore so its just placment that is checking | |
| 15:33:41 | jamespage | sean-k-mooney: having read most of the code that was my hunch so thanks for confirming... | |
| 15:34:09 | sean-k-mooney | so ya that is why we say you cant mix numa and non numa today | |
| 15:34:23 | sean-k-mooney | if we ever get to tracking numa in placment then it will fix that | |
| 15:34:43 | sean-k-mooney | sicne we will have a seperate pool per page size | |
| 15:34:54 | sean-k-mooney | and or numa node | |
| 15:36:21 | sean-k-mooney | we proably could paper over this temporally with a filter similar to the nuam one or even enhance the numa one to work with non numa instance but right now we just tell people not ot do it | |
| 15:42:29 | artom | I guess another option would be to tell operators to include the amount of hugepages in reserved_host_ram? | |
| 15:42:34 | artom | It's... weird, but should work? | |
| 15:42:51 | sean-k-mooney | no | |
| 15:43:09 | sean-k-mooney | it will break schduling | |
| 15:43:10 | artom | Wouldn't it just prevent scheduling to a host if it doesn't have enough "normal" RAM? | |
| 15:43:44 | sean-k-mooney | yes but you vould not use the hugepages and normal ram | |
| 15:44:03 | artom | Oh, right | |
| 15:44:14 | sean-k-mooney | you coul only use total - reserved | |
| 15:44:49 | sean-k-mooney | so that is what you do if you are using the hugepage on the host | |
| 15:45:07 | sean-k-mooney | + the huge page reserved option | |
| 15:50:52 | sean-k-mooney | so we coudl set max allocation size in placement = to total -hugepages | |
| 15:51:08 | sean-k-mooney | that would help | |
| 15:52:11 | jamespage | side effect to that would be that you could only ever allocate total - hugepages on that hypervisor no? | |
| 15:52:29 | sean-k-mooney | no | |
| 15:53:29 | jamespage | oh that's the max allocation size - driving the maximum single instance footprint - I see | |
| 15:53:45 | sean-k-mooney | https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-inventory-detail#update-resource-provider-inventory | |
| 15:53:48 | sean-k-mooney | yes | |
| 15:53:51 | sean-k-mooney | max_unit | |
| 15:54:22 | sean-k-mooney | sorry was just on the phone | |
| 15:54:31 | sean-k-mooney | so ya total woudl be set to total memroy | |
| 15:54:37 | sean-k-mooney | reserved = reserved form config | |
| 15:55:09 | sean-k-mooney | but if we set max_unit to total-reserved-hugepages that should help | |
| 15:55:35 | sean-k-mooney | although it would have to be the larger of | |
| 15:55:49 | sean-k-mooney | total-reserved-hugepages and hugepages | |
| 15:56:30 | sean-k-mooney | to not limit the size of hugepage guests | |
| 15:56:51 | sean-k-mooney | jamespage: is it a resent release | |
| 15:57:07 | sean-k-mooney | if so im not sure if you can use provider.yaml to twaak the make unit | |
| 15:57:26 | sean-k-mooney | gibi: do you rememebr if that can tweak non custom resouce classes? | |
| 15:57:55 | jamespage | ussuri for this particular deploy | |
| 15:58:50 | gibi | Only CUSTOM_* resource classes and traits may be managed this way. | |
| 15:58:56 | gibi | so no | |
| 15:59:00 | gibi | https://docs.openstack.org/nova/latest/admin/managing-resource-providers.html | |
| 15:59:07 | sean-k-mooney | ya ok | |
| 15:59:13 | sean-k-mooney | it was afetr ussuri anyway | |
| 15:59:51 | sean-k-mooney | jamespage: best we likely could do is a config option that could posible be backported to set or clamp the max unit | |
| 16:00:16 | sean-k-mooney | or just not mix for now | |
| 16:00:17 | gibi | we merged in victoria | |
| 16:01:51 | sean-k-mooney | jamespage: sorry i cant really be more help there. you could write a custom filter to do the same thing out of tree and load that in the env since its plugable | |
| 16:02:36 | jamespage | sean-k-mooney: no worries - you have been more that helpful in getting my knowledge up to speed in this area :) | |
| 16:02:36 | sean-k-mooney | for existign deployment that is the only way to make mixing numa/non numa on the same host somewhat ok but it still risks OOM events | |
| 16:03:12 | sean-k-mooney | ok i need to go do some other things and my home insure is not done so ill be afk now until monday | |
| 16:03:24 | sean-k-mooney | enjoy your weekends folks o/ | |
| 16:04:17 | gibi | o/ | |
| 16:51:35 | gibi | chateaulav: sorry I run out of time today. I will try to look at the issue tomorrow | |
| 16:51:38 | gibi | o/ | |
| 16:52:38 | chateaulav | thats sounds good, i think im in the right place and have been playing around with the tests for compute node. | |
| 18:33:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: WIP Improve remote-managed port test coverage https://review.opendev.org/c/openstack/nova/+/829974 | |
| 18:34:25 | opendevreview | Rajat Dhasmana proposed openstack/nova master: WIP: Add support for volume backed server rebuild https://review.opendev.org/c/openstack/nova/+/820368 | |
| 21:03:52 | opendevreview | Artom Lifshitz proposed openstack/nova master: block_device_info: Add swap to inline https://review.opendev.org/c/openstack/nova/+/826523 | |
| 21:03:53 | opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Improve creating images INFO log https://review.opendev.org/c/openstack/nova/+/826524 | |
| 21:03:53 | opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Remove defunct comment https://review.opendev.org/c/openstack/nova/+/826525 | |
| 21:03:54 | opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: default by_name image_type to config correctly https://review.opendev.org/c/openstack/nova/+/826526 | |
| 21:03:54 | opendevreview | Artom Lifshitz proposed openstack/nova master: image_meta: Add ephemeral encryption properties https://review.opendev.org/c/openstack/nova/+/760454 | |
| 21:03:55 | opendevreview | Artom Lifshitz proposed openstack/nova master: BlockDeviceMapping: Add encryption fields https://review.opendev.org/c/openstack/nova/+/760453 | |
| 21:03:55 | opendevreview | Artom Lifshitz proposed openstack/nova master: BlockDeviceMapping: Add is_local property https://review.opendev.org/c/openstack/nova/+/764485 | |
| 21:03:56 | opendevreview | Artom Lifshitz proposed openstack/nova master: compute: Update bdms with ephemeral encryption details when requested https://review.opendev.org/c/openstack/nova/+/764486 | |
| 21:03:56 | opendevreview | Artom Lifshitz proposed openstack/nova master: virt: Add ephemeral encryption flag https://review.opendev.org/c/openstack/nova/+/760455 | |
| 21:03:58 | opendevreview | Artom Lifshitz proposed openstack/nova master: scheduler: Add an ephemeral encryption pre filter https://review.opendev.org/c/openstack/nova/+/760456 | |
| 21:03:58 | opendevreview | Artom Lifshitz proposed openstack/nova master: block_device: Add DriverImageBlockDevice to block_device_info https://review.opendev.org/c/openstack/nova/+/826527 | |
| 21:04:00 | opendevreview | Artom Lifshitz proposed openstack/nova master: block_device: Add encryption attributes to image and ephemeral disks https://review.opendev.org/c/openstack/nova/+/826528 | |
| 21:04:00 | opendevreview | Artom Lifshitz proposed openstack/nova master: virt: Add block_device_info helper to find encrypted disks https://review.opendev.org/c/openstack/nova/+/826529 | |
| 21:04:02 | opendevreview | Artom Lifshitz proposed openstack/nova master: blockinfo: Add encryption details to the disk_info mappings when provided https://review.opendev.org/c/openstack/nova/+/772272 | |
| 21:04:02 | opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: Add disk_info_mapping as an optional attribute of Image https://review.opendev.org/c/openstack/nova/+/826530 | |
| 21:04:04 | opendevreview | Artom Lifshitz proposed openstack/nova master: privsep: Move qemu-img create calls under nova.privsep.qemu https://review.opendev.org/c/openstack/nova/+/826750 | |
| 21:04:04 | opendevreview | Artom Lifshitz proposed openstack/nova master: privsep: Return QemuImgInfo objects from qemu-img info calls https://review.opendev.org/c/openstack/nova/+/826751 | |
| 21:04:06 | opendevreview | Artom Lifshitz proposed openstack/nova master: privsep: Add encryption support to qemu-img create command https://review.opendev.org/c/openstack/nova/+/826752 | |
| 21:04:06 | opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Report ephemeral encryption traits based on imagebackend https://review.opendev.org/c/openstack/nova/+/826753 | |
| 21:04:08 | opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 21:04:08 | opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 21:04:10 | opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 21:04:10 | opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 22:02:32 | opendevreview | Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369 | |
| 22:02:32 | opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 22:02:33 | opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | |
| #openstack-nova - 2022-02-19 | |||
| 06:09:24 | opendevreview | Ghanshyam proposed openstack/nova master: Complete phase-1 of RBAC community-wide goal https://review.opendev.org/c/openstack/nova/+/829866 | |