| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-19 | |||
| 17:50:56 | sean-k-mooney | aspiers: hard_limit does not guarntee teh meoeoy will not be swapped or that it will be preallocated | |
| 17:51:37 | sean-k-mooney | aspiers: thos poperties are conntoeld via allocation and locked in the memory backing | |
| 17:53:20 | bbobrov | lets just ask on the libvirt mailing list | |
| 17:53:31 | bbobrov | (hi) | |
| 17:53:37 | aspiers | I agree with bbobrov | |
| 17:53:46 | aspiers | this probably requires libvirt changes | |
| 17:54:35 | aspiers | bbobrov: Do you want to kick off the thread there? | |
| 17:54:39 | sean-k-mooney | bbobrov: aspiers sure let us know what they say | |
| 17:55:25 | bbobrov | aspiers: yep, i will do it, but starting tomorrow | |
| 17:55:28 | aspiers | we can just link to the spec review and say "please find us a solution" :-) | |
| 17:55:33 | aspiers | bbobrov: great thanks | |
| 17:55:33 | sean-k-mooney | bbobrov: aspiers but as i said the behavior that is being asserted for hard_limit. i.e. that it prevents the page form migrating is not documented in teh libvirt docs https://libvirt.org/formatdomain.html#elementsMemoryTuning | |
| 17:55:42 | aspiers | sean-k-mooney: ack | |
| 17:56:16 | aspiers | https://libvirt.org/formatdomain.html#elementsMemoryBacking seems to be the bit for pinning but if that just does an mlock() then apparently it's not enough | |
| 17:56:23 | aspiers | from what I heard earlier | |
| 17:56:36 | aspiers | although that seems strange to me, since then why would it be good enough for realtime? | |
| 17:56:57 | aspiers | sean-k-mooney: real-time also needs a hard guarantee, right? | |
| 17:57:06 | bbobrov | why have we decided to use memtune for pinning in the first place? | |
| 17:57:22 | aspiers | because that's what the libvirt/QEMU guys recommended initially | |
| 17:57:27 | bbobrov | if there is no direct indication that it actually pins memory | |
| 17:57:35 | aspiers | they said it did | |
| 17:57:40 | bbobrov | is there a mailing list thread about it? | |
| 17:57:47 | aspiers | yes, on the internal list | |
| 17:58:02 | sean-k-mooney | aspiers: not entirly. for minium latency variance yes but it wont break the feature if there was host page migration in the background but it may break SLAs | |
| 17:58:08 | bbobrov | eh, internal lists | |
| 17:58:20 | aspiers | bbobrov: our sev list | |
| 17:58:39 | bbobrov | yeah, i got it, still ba | |
| 17:58:41 | bbobrov | *bad | |
| 17:59:10 | sean-k-mooney | bbobrov: well this is what the amdese repo does https://github.com/AMDESE/AMDSEV/blob/master/xmls/sample.xml#L218 | |
| 17:59:17 | sean-k-mooney | it jsue set hard_limit | |
| 17:59:57 | sean-k-mooney | there is no other memory tuning or pinning but i would not have assumed that pinned any memory | |
| 18:00:43 | aspiers | "The value of the domain/memtune/hard_limit element will be used to setrlimit(RLIMIT_MEMLOCK, hard_limit) on the qemu process and in memory.limit_in_bytes setting of the processes memory controller (/sys/fs/cgroup/memory/machine.slice/machine-<vmid><vmname>.scope/memory.limit_in_bytes)" | |
| 18:00:56 | aspiers | that's from our libvirt guy | |
| 18:01:28 | aspiers | also https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_domain.c;h=ba3fff607a93533b9b47956cc2cfa70237e7c041;hb=HEAD#l10134 | |
| 18:02:35 | gibi | mriedem: I've replied in https://review.openstack.org/#/c/640390/5/doc/source/admin/config-qos-min-bw.rst@98 | |
| 18:07:26 | bryan_stephenson | Accoring to https://libvirt.org/formatdomain.html#elementsMemoryBacking setting "locked" reserves/locks the physical memory. One use is for DMA, which would only be useful if the memory did not move. Do we think that memory might move locations even if it is locked? | |
| 18:07:49 | aspiers | bryan_stephenson: that's what Brijesh said on the call. he said mlock(2) was just a hint | |
| 18:08:02 | aspiers | the man page makes it sound like more than that, but I don't know | |
| 18:08:15 | bryan_stephenson | Then how does DMA not get goofed up? | |
| 18:08:33 | sean-k-mooney | bryan_stephenson we use hugepage memory for dma trasfer with dpdk so i think either hugepage or setting locked expclitly would be suffient | |
| 18:08:45 | aspiers | Honestly it's beyond my paygrade ;-) | |
| 18:09:20 | bryan_stephenson | So the "locked" just calls mlock() which may not be sufficient. Is that a correct understanding? | |
| 18:09:34 | aspiers | That's what I think I heard from Brijesh | |
| 18:12:06 | sean-k-mooney | aspiers: so looking at that funtion it runtrun Unlimited if the memory is locked to this would not caluatle a valide limit for realtime guests | |
| 18:12:08 | sean-k-mooney | https://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_domain.c;h=ba3fff607a93533b9b47956cc2cfa70237e7c041;hb=HEAD#l10049 | |
| 18:12:28 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add request-filter-image-types spec https://review.openstack.org/644625 | |
| 18:14:03 | efried | aspiers: Want me to throw a procedural -2 on the bottom SEV patch so you don't have to keep chasing -Ws? | |
| 18:14:18 | openstackgerrit | Merged openstack/nova master: Remove additional policy configuration details from policy doc https://review.openstack.org/644423 | |
| 18:14:25 | openstackgerrit | Merged openstack/nova master: Clarify policy shortcomings in policy enforcement doc https://review.openstack.org/643960 | |
| 18:14:31 | sean-k-mooney | aspiers: so based on https://github.com/libvirt/libvirt/blob/0ec6343a069b21178d4580688a8380dbb6d76620/docs/news-2013.html.in#L1526 | |
| 18:14:56 | sean-k-mooney | aspiers: libvirt used to set RLIMIT_MEMLOCK when you set the locked value | |
| 18:14:59 | aspiers | efried: that would be nice thanks, but depends on which you think is the bottom patch ;-) | |
| 18:15:08 | efried | aspiers: https://review.openstack.org/#/c/633855/ ? | |
| 18:15:19 | sean-k-mooney | aspiers: and it still does https://github.com/libvirt/libvirt/blob/b6aacfc435ce3b7e2665ddf5a422d2153bca88b8/src/util/virprocess.c#L749 | |
| 18:15:49 | efried | aspiers: Done, lmk if there are any stragglers not in the series that you'd like similarly tagged. | |
| 18:15:57 | aspiers | efried: yeah, I think 644554 could maybe merge pre-Stein without doing harm | |
| 18:16:05 | efried | agree | |
| 18:16:05 | aspiers | sean-k-mooney: reading | |
| 18:16:21 | efried | aspiers: if you can get that efried guy to approve it. | |
| 18:16:25 | aspiers | :) | |
| 18:16:33 | aspiers | efried: sounds like a tall order | |
| 18:17:19 | efried | aspiers: I almost did a full rewrite of that method, but I'll settle for nixing backslashes. | |
| 18:18:11 | aspiers | hrm, nix them how? | |
| 18:18:57 | efried | aspiers: Use parens, or use an if, or... | |
| 18:19:29 | aspiers | OK | |
| 18:20:04 | efried | it's just a style rule that nova likes to follow. I'm pretty sure there was a time I didn't care, but now I'm brainwashed. | |
| 18:20:09 | aspiers | sure | |
| 18:22:50 | aspiers | sean-k-mooney: seems like a key question is whether mlock(2) really is enough for SEV or not | |
| 18:26:01 | aspiers | sean-k-mooney, bbobrov: I guess there's also #libvirt | |
| 18:30:16 | sean-k-mooney | aspiers: i think its calling https://linux.die.net/man/2/setrlimit but yes let use know what they determin | |
| 18:33:24 | sean-k-mooney | but as i said before im pretry sure hard_limit is not actully locking the memory its just limiting how much memory can be locked by an unprivalaged process | |
| 18:33:58 | aspiers | sean-k-mooney: yes, I think it needs the combination of setrlimit and mlock | |
| 18:35:18 | sean-k-mooney | which https://github.com/AMDESE/AMDSEV/blob/master/xmls/sample.xml does not do :) | |
| 18:35:44 | sean-k-mooney | anyway im going to grab something to eat | |
| 18:35:49 | sean-k-mooney | be back in a bit | |
| 18:35:57 | aspiers | OK, me too. Cheers for the help! | |
| 18:49:28 | openstackgerrit | Lance Bragstad proposed openstack/nova master: Give the policy vision document a facelift https://review.openstack.org/644615 | |
| 19:30:02 | sean-k-mooney | /away until noon gmt | |
| 19:30:19 | sean-k-mooney | extra space .... | |
| 19:45:54 | spbarbieri | I provision a server with a block_device that specifies a volume to be used as the boot volume. I'd like to use the REST API to determine if a block_device was used to boot the instance, and if so, which attached volume is the boot volume for the server. The "server details" api returns an empty "image" so I guess I can assume it used a volume as the boot device, but I don't know how to determine which attached volume is t | |
| 19:50:58 | artom | spbarbieri, you can list volume attachments with https://developer.openstack.org/api-ref/compute/?expanded=list-volume-attachments-for-an-instance-detail,show-a-detail-of-a-volume-attachment-detail#list-volume-attachments-for-an-instance, but that wouldn't tell you whether one was used for boot | |
| 19:58:45 | spbarbieri | Thanks for the pointer. When creating the server I can specify "block_device_mapping_v2.delete_on_termination". This is saved somewhere because my volume is deleted on server termination. I was hoping that the "this is the boot device" was saved there as well. But, I couldn't find it in any of the server/volume/volume_attach apis. | |
| 20:05:47 | artom | spbarbieri, I *think* boot_index is what you're looking for, but it doesn't look like we expose it anywhere in the API | |
| 20:06:03 | artom | spbarbieri, you can also tag with 'boot' as a workaround | |
| 20:14:22 | spbarbieri | yeah, boot_index would be perfect. I couldn't find it anywhere either. I'm actually trying to do this after the fact. I have a running server and I want to know which volume was used as the boot drive. I only have access to the REST API...sounds like a MacGyver episode...find the boot drive using a paperclip and some duct tape | |
| 20:23:05 | openstackgerrit | Merged openstack/nova master: Use assertXmlEqual() helper for all XML comparison tests https://review.openstack.org/641852 | |
| 20:23:13 | openstackgerrit | Merged openstack/nova stable/rocky: Allow utime call to fail on qcow2 image base file https://review.openstack.org/643011 | |
| 20:24:00 | mriedem | spbarbieri: that's eventually going to show up in the API here https://review.openstack.org/#/c/623981/24 | |
| 20:24:13 | mriedem | e.g. https://review.openstack.org/#/c/623981/24/doc/api_samples/os-volumes/v2.72/volume-attachment-detail-resp.json | |
| 20:25:56 | mriedem | if you're an admin, you could also see the OS-EXT-SRV-ATTR:root_device_name parameter in the response from the GET /servers/{server_id} API | |
| 20:26:21 | mriedem | and match that to the volume with the same device name, although those aren't guaranteed to be where the root volume is mounted in the guest | |
| 20:27:58 | spbarbieri | Thanks for the pointer to the volume-attach details. That will be helpful. I'm running against my local devstack, as admin, and I don't get that returned. I'm calling compute/v2.1/servers/{id}. | |
| 20:55:06 | melwitt | spbarbieri: are you using novaclient or openstackclient? if the latter, you need to pass the necessary compute API microversion to see the volume-attach info. i.e. --os-compute-api-version 2.3 (see https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-kilo) | |
| 20:55:35 | melwitt | s/volume-attach info/root_device_name/ | |
| 20:57:51 | spbarbieri | melwitt: I'm using Postman with direct REST API calls | |
| 21:00:28 | melwitt | spbarbieri: OK, see this doc on how to specify compute API microversion with the HTTP header https://developer.openstack.org/api-guide/compute/microversions.html#client-interaction | |
| 21:04:41 | spbarbieri | melwitt: Yes! Thanks. Adding that header works. I guess now I can match the root_device_name against all of the attached volumes to find a match. Does that seem reasonable? | |
| 21:06:24 | melwitt | spbarbieri: I think so, based on what mriedem said earlier | |
| 21:07:07 | spbarbieri | melwitt: Thank you for the help. | |
| 21:07:18 | melwitt | np, good luck | |