| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-19 | |||
| 13:33:51 | kashyap | Hard-coding a memory value. | |
| 13:34:11 | sean-k-mooney | ya but we dont need to do that if we use hugepages which i suggested last cycle | |
| 13:34:25 | kashyap | Also each AMD machine has some limit on the number of VMs. That is also something that needs to be worked into the spec | |
| 13:35:01 | sean-k-mooney | ya that is new. it was not raised previously but that is simple enough to fix with a inventory of sev context in placement | |
| 13:35:20 | sean-k-mooney | assuming we can auto discover that limit form either libvirt or sysfs | |
| 13:35:54 | kashyap | I don't think this is just limited to hugepages. The core point about protecting the host from getting DoSed needs to be addressed. | |
| 13:36:00 | sean-k-mooney | if we really needed too we could have a config option for it. that shoudl be in the spec i agree but i still dont see a hard blocker here | |
| 13:36:02 | kashyap | Please see the cgroups idea DanPB mentioned in his remark. | |
| 13:36:59 | kashyap | I think this is hard blocker. I'm even more convinced of it (that's why I gave a "-1", FWIW) after reading DanPB's message and talking to a developer who implemented SEV bits in libvirt | |
| 13:37:16 | kashyap | Let's work it out on the spec. IRC is too real-time for me sometimes :-) | |
| 13:37:31 | sean-k-mooney | kashyap: well if you require the use of hugepage then you disable over commit and dont need to use a hard memory limit as the vm memory is preallcoated and fixed | |
| 13:37:51 | kashyap | sean-k-mooney: So you're saying when using SEV one should "mandate" huge pages? | |
| 13:37:56 | sean-k-mooney | yes | |
| 13:38:09 | kashyap | Hmm, I don't know the full implications of it. Need to think more. | |
| 13:38:13 | sean-k-mooney | i suggested it last cycle to not need to set the hardlimit | |
| 13:38:35 | sean-k-mooney | the main implication is no over commit | |
| 13:38:48 | sean-k-mooney | and an implict numa topology | |
| 13:39:36 | sean-k-mooney | the memory for sev i think i memlocked so it cant be swapped anyway so you would have disabled over commit already | |
| 13:39:59 | sean-k-mooney | so requireing hugepages would remove some of the complexity of this feature and give a performacne boost | |
| 13:40:10 | kashyap | Okay, I see the pinning the memory by using `hw:mem_page_size=large` point in the spec. | |
| 13:41:33 | mriedem | efried: cdent: edleafe: semi interesting but also for an older release bug about forced live migration failing during PUT /allocations https://bugs.launchpad.net/nova/+bug/1820612 | |
| 13:41:34 | openstack | Launchpad bug 1820612 in OpenStack Compute (nova) "Logging is hard to read if there is a problem with resources during live migration" [Undecided,Incomplete] | |
| 13:41:35 | kashyap | sean-k-mooney: Please respond to Dan on the review, for the discussion to be "sticky". | |
| 13:41:42 | sean-k-mooney | ya i didnt push too hard because it would be nice to not require hugepages but it make this a hole lot eaiser as not useing htem has other accounting issues | |
| 13:41:58 | mriedem | efried: cdent: edleafe: i've pieced together that they are forcing a live migration to a dest host before queens where we'd move the source node allocations from the instance to the migration record, | |
| 13:42:20 | sean-k-mooney | kashyap: sure. | |
| 13:42:21 | mriedem | so in this code, it's doubling up the allocations for the instance consumer by putting the same source node allocations to the source provider and the new allocations to the dest provider, | |
| 13:42:31 | mriedem | but placement pukes on the source provider PUT | |
| 13:42:52 | mriedem | which makes me wonder if placement is trying to make a new allocation rather than replace it | |
| 13:43:48 | mdbooth | This monkey patching change is a hydra :( | |
| 13:45:01 | mriedem | cdent: anyway if they were queens+ they wouldn't have this problem, but that isn't a very helpful response to someone at ocata or pike | |
| 13:45:18 | mriedem | if we know about a bug fix in placement though i could say 'yup this is fixed here, backport it' | |
| 13:45:21 | kashyap | mdbooth: Seems like it is turning into "cthulhu patching" | |
| 13:45:26 | cdent | mriedem: I'm still digesting. | |
| 13:45:32 | edleafe | mriedem: I don't remember ever doubling the resources on the source when migrating to a different host | |
| 13:46:01 | mdbooth | kashyap: Eventlet is just awful. | |
| 13:46:50 | kashyap | mdbooth: Nod, I only hear about it (I didn't have to faff with it much so far). Skimmed your e-mail; thanks for the write-up! | |
| 13:46:58 | mdbooth | I still don't understand what problem it solves, tbh. Seems to me it just makes your multi-threading even jankier. | |
| 13:47:50 | edleafe | mdbooth: One of my first tasks in OpenStack was to replace the Twisted code with eventlet, because people were having so much trouble tracking down bugs in Twisted :) | |
| 13:47:51 | kashyap | mdbooth: I saw your second email about using "Python's APIs except when not possible" for multi-threading | |
| 13:48:08 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add detection of SEV support from QEMU/AMD-SP/libvirt on AMD hosts https://review.openstack.org/633855 | |
| 13:48:26 | mdbooth | edleafe: Hah. There's always something worse :) | |
| 13:48:35 | edleafe | mdbooth: Yup | |
| 13:48:53 | mdbooth | edleafe: But at a top level, why not just use python's multi-threading? | |
| 13:49:32 | cdent | mriedem: Is there any chance there's a race. Something else ate a bit more more RAM during the transaction. That shouldn't be possible (at least not in more recent allocation handling) but it's very hard to keep track of changes... | |
| 13:49:51 | mdbooth | Python already had fake concurrency, so why implement fake concurrency on top of fake concurrency? | |
| 13:49:51 | edleafe | mdbooth: I really don't remember. The original nova code used Twisted for async, and it caused a lot of problems, so someone suggested eventlet, and... | |
| 13:49:58 | kashyap | aspiers: Thanks for addressing the docs bit. You saw what I "discovered" with my comment :D Writing that out helped me understand it better | |
| 13:50:15 | mriedem | cdent: it's possible, this was pre-generations | |
| 13:50:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup comments around claim_resources method https://review.openstack.org/644576 | |
| 13:51:12 | mriedem | cdent: i guess one way to find out is write a functional regression test with at-capacity source compute and then try to live migrate off it | |
| 13:51:26 | mriedem | tests against at-capacity computes is not our strong suit in nova | |
| 13:51:32 | cdent | quite | |
| 13:51:51 | mriedem | anyway, while in here i've cleaned up a bunch of old comments https://review.openstack.org/644576 | |
| 13:52:23 | cdent | huzzah | |
| 13:54:32 | sean-k-mooney | aspiers: kashyap i replied in the spec reproposal | |
| 13:55:45 | aspiers | kashyap: yep, that extra clarity helped me too. Have just replied inline in the review | |
| 13:55:54 | aspiers | sean-k-mooney: thanks | |
| 14:04:54 | openstackgerrit | Adam Spiers proposed openstack/nova master: Add new "supports_amd_sev" capability to libvirt driver https://review.openstack.org/638680 | |
| 14:05:21 | mriedem | alex_xu: replied in https://review.openstack.org/#/c/644293/ - thanks for those questions | |
| 14:05:51 | efried | mriedem: This ready to go? https://review.openstack.org/#/c/644423/ I didn't want to rubber-stamp it if you wanted another look. | |
| 14:06:07 | mriedem | efried: looking | |
| 14:06:25 | mriedem | got it | |
| 14:07:04 | efried | thx | |
| 14:12:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add docs for compute capabilities as traits https://review.openstack.org/644293 | |
| 14:12:32 | mriedem | efried: aspiers: alex_xu: ^ updated to use one of aspiers' suggestions and document the resource_provider_association_refresh thing in the commit message | |
| 14:14:05 | efried | mriedem: +2, still looks good | |
| 14:20:46 | kashyap | aspiers: I updated the machine types blueprint name (to "gracefully-handle-qemu-machine-types"). When I originally wrote it, discussions were hot on QEMU / libvirt & related lists about making 'q35' default. But since then many things have changed. | |
| 14:21:03 | kashyap | aspiers: When you next respin the patch, might want to update the reference to it. | |
| 14:38:17 | kashyap | aspiers: For later discussion, I'm still thinking if Nova should be in the business of deciding the default machine type fo 'x86'. I'm updating the problem description in the specification. Can discuss there | |
| 14:38:24 | kashyap | (So feel free to disregard me here.) | |
| 14:41:58 | sean-k-mooney | kashyap: right now you can set it in the config if you chooes to otherwise we leave it up to qemu | |
| 14:42:29 | sean-k-mooney | kashyap: im not sure if we need to chage that or not as intaller tool like tripleO can always just set it in the config | |
| 14:42:48 | kashyap | sean-k-mooney: Yes, I'm writing it up in the spec, if only for discussion, and that we considered it. | |
| 14:42:59 | kashyap | (There was an elaborate discussion on the 'qemu-devel' and 'libvirt' lists some months ago to change the defaults.) | |
| 14:43:16 | sean-k-mooney | yes i know :) | |
| 14:43:17 | kashyap | People seem to be overly paranoid about "backwards compat". | |
| 14:43:38 | sean-k-mooney | well it would break centos 6 vms | |
| 14:43:45 | sean-k-mooney | which are apparently still a thing | |
| 14:44:29 | sean-k-mooney | but ya i generally say use a sane default and if there isnt one leave it up to the operator | |
| 14:44:47 | aspiers | kashyap: ah, got it thanks | |
| 14:45:16 | sean-k-mooney | you could argue q35 is a sane default on security grounds but backwards comptiablity is also a good argument against it at least for a time | |
| 14:45:50 | dansmith | mriedem: yeah, you marked it as wishlist...does it need further action to demote it? | |
| 14:46:04 | sean-k-mooney | eventrualy security will win and we will want to change the default. not sure if its now or not but its good to discusse in anycase | |
| 14:52:20 | mriedem | dansmith: if you think it's a "never going to happen" thing, then you could mark it as won't fix with further words | |
| 14:52:52 | mriedem | dansmith: also for your awareness, https://review.openstack.org/#/c/640688/ is being discussed by the tc | |
| 14:54:11 | openstackgerrit | Matthew Booth proposed openstack/nova master: Eventlet monkey patching should be as early as possible https://review.openstack.org/626952 | |
| 14:54:39 | dansmith | mriedem: I don't know that I have a strong opinion on that (nor do I know why the tc would be involved in a discussion about it, but...) | |
| 14:54:40 | kashyap | Anyone with an Ubuntu machine, can you please post this output in a pastebin: `qemu-system-x86 -machine help` | |
| 14:56:03 | dansmith | mriedem: AFAICT, that's nova publishing metrics for which it is the authority to an external metrics-gathering service | |
| 14:56:39 | dansmith | mriedem: I've previously been opposed to abusing nova to collect metrics on non-nova things "since it's already deployed on the compute node" instead of installing a proper metrics-gathering daemon | |
| 14:56:45 | dansmith | but this is not that thing | |
| 14:56:47 | mriedem | kashyap: paste.openstack.org/show/748029/ | |
| 14:56:55 | mriedem | http://paste.openstack.org/show/748029/ | |
| 14:57:05 | kashyap | mriedem: Thank you! | |
| 14:58:08 | mriedem | dansmith: ack, see my comments about the smorgasbord of existing half baked metrics/usage/audit things we already expose | |
| 14:58:16 | mriedem | my main opposition is piling on to that pile | |
| 14:58:28 | dansmith | yeah, this is not that, AFAICT | |
| 15:11:08 | openstackgerrit | sahid proposed openstack/os-traits master: ci: re-enable openstack-python35-jobs https://review.openstack.org/644595 | |
| 15:12:19 | openstackgerrit | Gorka Eguileor proposed openstack/nova master: Use os-brick locking for volume attach and detach https://review.openstack.org/614190 | |