| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-15 | |||
| 14:40:00 | sean-k-mooney | alex_xu: checking avilable memory is checking free memory so it decresase with each vm booted | |
| 14:40:59 | sean-k-mooney | so if i have 20 gigabes of memory and ram allocation raitio of 1.5 avilable will alow only 2 10G vms to both and total will allow 3 respecting the allocation ratio | |
| 14:41:04 | alex_xu | sean-k-mooney: yea, that is we said not allow overcommit by checking free memory, right? | |
| 14:41:36 | sean-k-mooney | alex_xu: over commit of ram is intended to be support for pinned instances | |
| 14:41:53 | alex_xu | sean-k-mooney: ah...why | |
| 14:42:08 | sean-k-mooney | stephenfin: if you chage it to patial bug then it can be a follow up but if you want it to be closed bug then it needs to be in this change | |
| 14:42:28 | alex_xu | then I would say that isn't the problem of stephenfin's patch | |
| 14:42:42 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Restore nova-consoleauth to install docs https://review.openstack.org/630997 | |
| 14:42:49 | stephenfin | sean-k-mooney: You sure? It's very much a different bug | |
| 14:42:50 | sean-k-mooney | alex_xu: the cpu pinning specs never forbid over subsription of memory and it was a bevhioral change that was intoduced in error after the fact with no spec | |
| 14:43:19 | sean-k-mooney | stephenfin: its not from my point of view its the exact same bug | |
| 14:43:23 | alex_xu | sean-k-mooney: ok, got it | |
| 14:44:09 | alex_xu | so that is another thing now | |
| 14:45:27 | sean-k-mooney | stephenfin: if you update the bug to specicialy state it dose not fix pinned instance and file a new bug then you could close the existing bug. i consider pinned instance to just be a special case of instnace with numa topologies | |
| 14:46:21 | alex_xu | whatever...close to 11pm, I need to go to bed now | |
| 14:47:56 | sean-k-mooney | alex_xu: good night o/ | |
| 14:49:17 | maciejjozefczyk | stephenfin: hey, if you have a minute PTAL https://review.openstack.org/#/c/614167/ and https://review.openstack.org/#/c/591607. You have already reviewed that :) Thanks | |
| 14:50:14 | stephenfin | maciejjozefczyk: Just working through the above hugepage issues atm, but I'll stick it on my review queue :) | |
| 14:50:37 | maciejjozefczyk | stephenfin: good luck :) | |
| 14:53:33 | sean-k-mooney | stephenfin: ill reply to the bug and suggest coverign the pinning issue in a follwo up patch | |
| 14:53:47 | sean-k-mooney | as a seperate bug | |
| 14:56:41 | stephenfin | sean-k-mooney: I'm rather confused now http://paste.openstack.org/show/742644/ | |
| 14:58:57 | sean-k-mooney | looking | |
| 15:00:37 | sean-k-mooney | host_cell.avail_memory include hugepages | |
| 15:01:08 | sean-k-mooney | so the first instace boots | |
| 15:01:19 | sean-k-mooney | and the second lets see | |
| 15:01:20 | stephenfin | yup, I get that. What's up with the second one though | |
| 15:01:31 | stephenfin | I figured this would prevent that from happening https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1021-L1036 | |
| 15:02:05 | ShilpaSD | mriedem: Hi, replied to your comment and uploaded patch https://review.openstack.org/#/c/612626/2, Please give your feedback on the latest patch. | |
| 15:02:24 | sean-k-mooney | stephenfin: is this a dual numa system | |
| 15:02:52 | stephenfin | shouldn't matter. I've configured the same amount of hugepages for both, and both sockets appear to have ~ the same amount of memory | |
| 15:03:02 | sean-k-mooney | i see form the free you have 6.7G free but the cat command shows ~400MB free so im guessing its booting on numa node 1 | |
| 15:03:32 | sean-k-mooney | stephenfin: you have configred 16GB of hugepages total | |
| 15:03:50 | stephenfin | sean-k-mooney: Yup, 8GB per node | |
| 15:04:02 | sean-k-mooney | so there are 8G of small pages on each node some of which are used by the os | |
| 15:04:26 | sean-k-mooney | oh its a 10G vm not 1G | |
| 15:04:31 | stephenfin | Yup | |
| 15:04:36 | stephenfin | Sorry, should have highlighted that | |
| 15:05:04 | stephenfin | 16 GB per node - 8 GB 2M hugepages should leave only 8 GB usable small pages per node | |
| 15:05:05 | sean-k-mooney | no i just missed the 0 at the end | |
| 15:05:13 | sean-k-mooney | yep | |
| 15:05:22 | stephenfin | So a 10 GB instance should fall flat on its face | |
| 15:05:48 | sean-k-mooney | so rather then looking at this output its better to look at libvirts | |
| 15:06:18 | sean-k-mooney | also what is your ram allocation ratio set to im assuming 1.5 | |
| 15:06:40 | stephenfin | 16.0 | |
| 15:06:42 | stephenfin | manually set | |
| 15:06:50 | stephenfin | shouldn't matter for hugepages though | |
| 15:06:50 | sean-k-mooney | your ram is set to 16 | |
| 15:06:54 | sean-k-mooney | true | |
| 15:07:15 | sean-k-mooney | do you have your patch applied or master | |
| 15:07:31 | stephenfin | my patch | |
| 15:09:51 | sean-k-mooney | with an explicit pagesize of small we expect https://github.com/openstack/nova/blob/89c2d1056b3f37bad243c7468d974ded71336a0f/nova/virt/hardware.py#L623-L649 to reject it correct | |
| 15:10:52 | stephenfin | Yes, that would be my expectation | |
| 15:11:25 | sean-k-mooney | so you have not modifed that in your patch so that is either broken and has always been or there is something else going on | |
| 15:12:03 | sean-k-mooney | well you changed the name of the fuction it calls but otherwise its the same | |
| 15:27:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: The field instance_name was added to InstanceCreatePayload https://review.openstack.org/627967 | |
| 15:36:12 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Clean up versioned object backlevelling code https://review.openstack.org/630976 | |
| 15:40:14 | sean-k-mooney | stephenfin: looks like i was wrong about pinning and memory oversubsription. https://github.com/openstack/nova/commit/892e4032acea379bac6d1ae18fc5271ea17c0131 the intel implementation did not prevent it and the spec did not appove that behavior as a result we do not document it https://docs.openstack.org/nova/pike/admin/cpu-topologies.html#customizing-instance-cpu-pinning-policies. i consider that a | |
| 15:40:16 | sean-k-mooney | long term but which i think we should resolve seperatly | |
| 15:49:04 | mdbooth | cdent: Don't suppose you submitted that tox.ini patch, did you? | |
| 15:50:04 | mdbooth | cdent: NM, some gerrit-fu was a match to the task | |
| 15:52:27 | mriedem | lyarwood: is the conflict in here still valid? https://review.openstack.org/#/c/627958/ | |
| 15:53:40 | lyarwood | mriedem: yeah it's actually due to I217fba9138132b107e9d62895d699d238392e761 landing in stable/rocky after the initial PS I posted | |
| 15:53:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: libvirt: Add workaround to cleanup instance dir when using rbd https://review.openstack.org/627958 | |
| 15:54:43 | mriedem | lyarwood: yeah fixe | |
| 15:54:45 | mriedem | *fixed | |
| 15:55:01 | lyarwood | huh, I don't get it anymore, weird. | |
| 15:55:19 | lyarwood | okay well it wasn't clean just after that landed but is now, meh! | |
| 16:20:09 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: docs: Secure live migration with QEMU-native TLS https://review.openstack.org/629627 | |
| 16:20:10 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: A few miscellaneous items related to "native TLS" https://review.openstack.org/630980 | |
| 16:20:44 | kashyap | (Okay, let's put the bikeshed paint back in the cellar.) | |
| 16:22:00 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Clean up versioned object backlevelling code https://review.openstack.org/630976 | |
| 16:53:41 | cdent | mdbooth: did you find (and then try) that py37 tox fix? I was on the wrong machine to be able to give it a totally robust check | |
| 16:57:46 | mdbooth | cdent: Yes. Tested locally, all good. | |
| 16:57:54 | cdent | mdbooth: great, thanks | |
| 16:58:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Send context.global_id on neutron calls https://review.openstack.org/627034 | |
| 16:58:17 | mdbooth | cdent: I rebased my current work on top of it for convenience. | |
| 16:59:46 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Send context.global_id on neutron calls https://review.openstack.org/627603 | |
| 17:01:02 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Send context.global_id on neutron calls https://review.openstack.org/627604 | |
| 17:03:08 | mdbooth | cdent: It breaks my 'testone' script, though, by not using the 'py37' env any more :/ I note that's by design, though. | |
| 17:03:30 | cdent | tis | |
| 17:06:11 | cfriesen | mriedem: so I guess the "specify timeout per live migration request" spec kind of died on the vine? | |
| 17:20:43 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add a note in "nova service-delete" help about deleting computes https://review.openstack.org/627637 | |
| 17:39:21 | eandersson | It's still not possible to make weights plugins right? | |
| 17:42:58 | mriedem | stephenfin: does the latest version of https://review.openstack.org/#/c/623564/ fix your problem? | |
| 17:43:45 | stephenfin | mriedem: Knee deep in hugepage issues and don't want to unload that context. I can check first thing tomorrow though and post my feedback on the review | |
| 17:44:28 | sean-k-mooney | mriedem: that is in relate to sqlite | |
| 17:44:40 | sean-k-mooney | mriedem: i can test that locally with a docker container and comment back | |
| 17:44:55 | stephenfin | sean-k-mooney: *changes in sqlite highlighted by sqlalchemy-migrate | |
| 17:46:12 | sean-k-mooney | right so if i clone that change an pip install it into my test env i should be able to test it with sqlite 3.26 | |
| 17:46:45 | sean-k-mooney | so i can repoduce the bug locally again and manully test this since your busy | |
| 17:52:24 | jaypipes | gibi: sorry, when is the bw scheduling hangout? | |
| 17:53:08 | sean-k-mooney | jaypipes: ehterpad say jan 18th at 17:00 utc | |
| 17:53:14 | sean-k-mooney | https://etherpad.openstack.org/p/bandwidth-way-forward | |
| 17:53:15 | jaypipes | sean-k-mooney: danke | |
| 17:53:41 | sean-k-mooney | it was originally ment to be an hour ago but was moved | |
| 17:54:22 | jaypipes | ack, I remember it moved but couldn't remember the new time. now on my calendar. thx again | |
| 18:06:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix overcommit for NUMA-based instances https://review.openstack.org/629281 | |
| 18:06:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Handle libvirt reporting incorrect 4k page quantities https://review.openstack.org/631038 | |
| 18:06:32 | stephenfin | sean-k-mooney: ^ | |