| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-15 | |||
| 14:56:41 | stephenfin | sean-k-mooney: I'm rather confused now http://paste.openstack.org/show/742644/ | |
| 14:58:57 | sean-k-mooney | looking | |
| 15:00:37 | sean-k-mooney | host_cell.avail_memory include hugepages | |
| 15:01:08 | sean-k-mooney | so the first instace boots | |
| 15:01:19 | sean-k-mooney | and the second lets see | |
| 15:01:20 | stephenfin | yup, I get that. What's up with the second one though | |
| 15:01:31 | stephenfin | I figured this would prevent that from happening https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1021-L1036 | |
| 15:02:05 | ShilpaSD | mriedem: Hi, replied to your comment and uploaded patch https://review.openstack.org/#/c/612626/2, Please give your feedback on the latest patch. | |
| 15:02:24 | sean-k-mooney | stephenfin: is this a dual numa system | |
| 15:02:52 | stephenfin | shouldn't matter. I've configured the same amount of hugepages for both, and both sockets appear to have ~ the same amount of memory | |
| 15:03:02 | sean-k-mooney | i see form the free you have 6.7G free but the cat command shows ~400MB free so im guessing its booting on numa node 1 | |
| 15:03:32 | sean-k-mooney | stephenfin: you have configred 16GB of hugepages total | |
| 15:03:50 | stephenfin | sean-k-mooney: Yup, 8GB per node | |
| 15:04:02 | sean-k-mooney | so there are 8G of small pages on each node some of which are used by the os | |
| 15:04:26 | sean-k-mooney | oh its a 10G vm not 1G | |
| 15:04:31 | stephenfin | Yup | |
| 15:04:36 | stephenfin | Sorry, should have highlighted that | |
| 15:05:04 | stephenfin | 16 GB per node - 8 GB 2M hugepages should leave only 8 GB usable small pages per node | |
| 15:05:05 | sean-k-mooney | no i just missed the 0 at the end | |
| 15:05:13 | sean-k-mooney | yep | |
| 15:05:22 | stephenfin | So a 10 GB instance should fall flat on its face | |
| 15:05:48 | sean-k-mooney | so rather then looking at this output its better to look at libvirts | |
| 15:06:18 | sean-k-mooney | also what is your ram allocation ratio set to im assuming 1.5 | |
| 15:06:40 | stephenfin | 16.0 | |
| 15:06:42 | stephenfin | manually set | |
| 15:06:50 | sean-k-mooney | your ram is set to 16 | |
| 15:06:50 | stephenfin | shouldn't matter for hugepages though | |
| 15:06:54 | sean-k-mooney | true | |
| 15:07:15 | sean-k-mooney | do you have your patch applied or master | |
| 15:07:31 | stephenfin | my patch | |
| 15:09:51 | sean-k-mooney | with an explicit pagesize of small we expect https://github.com/openstack/nova/blob/89c2d1056b3f37bad243c7468d974ded71336a0f/nova/virt/hardware.py#L623-L649 to reject it correct | |
| 15:10:52 | stephenfin | Yes, that would be my expectation | |
| 15:11:25 | sean-k-mooney | so you have not modifed that in your patch so that is either broken and has always been or there is something else going on | |
| 15:12:03 | sean-k-mooney | well you changed the name of the fuction it calls but otherwise its the same | |
| 15:27:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: The field instance_name was added to InstanceCreatePayload https://review.openstack.org/627967 | |
| 15:36:12 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Clean up versioned object backlevelling code https://review.openstack.org/630976 | |
| 15:40:14 | sean-k-mooney | stephenfin: looks like i was wrong about pinning and memory oversubsription. https://github.com/openstack/nova/commit/892e4032acea379bac6d1ae18fc5271ea17c0131 the intel implementation did not prevent it and the spec did not appove that behavior as a result we do not document it https://docs.openstack.org/nova/pike/admin/cpu-topologies.html#customizing-instance-cpu-pinning-policies. i consider that a | |
| 15:40:16 | sean-k-mooney | long term but which i think we should resolve seperatly | |
| 15:49:04 | mdbooth | cdent: Don't suppose you submitted that tox.ini patch, did you? | |
| 15:50:04 | mdbooth | cdent: NM, some gerrit-fu was a match to the task | |
| 15:52:27 | mriedem | lyarwood: is the conflict in here still valid? https://review.openstack.org/#/c/627958/ | |
| 15:53:40 | lyarwood | mriedem: yeah it's actually due to I217fba9138132b107e9d62895d699d238392e761 landing in stable/rocky after the initial PS I posted | |
| 15:53:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: libvirt: Add workaround to cleanup instance dir when using rbd https://review.openstack.org/627958 | |
| 15:54:43 | mriedem | lyarwood: yeah fixe | |
| 15:54:45 | mriedem | *fixed | |
| 15:55:01 | lyarwood | huh, I don't get it anymore, weird. | |
| 15:55:19 | lyarwood | okay well it wasn't clean just after that landed but is now, meh! | |
| 16:20:09 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: docs: Secure live migration with QEMU-native TLS https://review.openstack.org/629627 | |
| 16:20:10 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: A few miscellaneous items related to "native TLS" https://review.openstack.org/630980 | |
| 16:20:44 | kashyap | (Okay, let's put the bikeshed paint back in the cellar.) | |
| 16:22:00 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Clean up versioned object backlevelling code https://review.openstack.org/630976 | |
| 16:53:41 | cdent | mdbooth: did you find (and then try) that py37 tox fix? I was on the wrong machine to be able to give it a totally robust check | |
| 16:57:46 | mdbooth | cdent: Yes. Tested locally, all good. | |
| 16:57:54 | cdent | mdbooth: great, thanks | |
| 16:58:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Send context.global_id on neutron calls https://review.openstack.org/627034 | |
| 16:58:17 | mdbooth | cdent: I rebased my current work on top of it for convenience. | |
| 16:59:46 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Send context.global_id on neutron calls https://review.openstack.org/627603 | |
| 17:01:02 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Send context.global_id on neutron calls https://review.openstack.org/627604 | |
| 17:03:08 | mdbooth | cdent: It breaks my 'testone' script, though, by not using the 'py37' env any more :/ I note that's by design, though. | |
| 17:03:30 | cdent | tis | |
| 17:06:11 | cfriesen | mriedem: so I guess the "specify timeout per live migration request" spec kind of died on the vine? | |
| 17:20:43 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add a note in "nova service-delete" help about deleting computes https://review.openstack.org/627637 | |
| 17:39:21 | eandersson | It's still not possible to make weights plugins right? | |
| 17:42:58 | mriedem | stephenfin: does the latest version of https://review.openstack.org/#/c/623564/ fix your problem? | |
| 17:43:45 | stephenfin | mriedem: Knee deep in hugepage issues and don't want to unload that context. I can check first thing tomorrow though and post my feedback on the review | |
| 17:44:28 | sean-k-mooney | mriedem: that is in relate to sqlite | |
| 17:44:40 | sean-k-mooney | mriedem: i can test that locally with a docker container and comment back | |
| 17:44:55 | stephenfin | sean-k-mooney: *changes in sqlite highlighted by sqlalchemy-migrate | |
| 17:46:12 | sean-k-mooney | right so if i clone that change an pip install it into my test env i should be able to test it with sqlite 3.26 | |
| 17:46:45 | sean-k-mooney | so i can repoduce the bug locally again and manully test this since your busy | |
| 17:52:24 | jaypipes | gibi: sorry, when is the bw scheduling hangout? | |
| 17:53:08 | sean-k-mooney | jaypipes: ehterpad say jan 18th at 17:00 utc | |
| 17:53:14 | sean-k-mooney | https://etherpad.openstack.org/p/bandwidth-way-forward | |
| 17:53:15 | jaypipes | sean-k-mooney: danke | |
| 17:53:41 | sean-k-mooney | it was originally ment to be an hour ago but was moved | |
| 17:54:22 | jaypipes | ack, I remember it moved but couldn't remember the new time. now on my calendar. thx again | |
| 18:06:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Handle libvirt reporting incorrect 4k page quantities https://review.openstack.org/631038 | |
| 18:06:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix overcommit for NUMA-based instances https://review.openstack.org/629281 | |
| 18:06:32 | stephenfin | sean-k-mooney: ^ | |
| 18:06:40 | stephenfin | alex_xu: too ^ | |
| 18:09:22 | sean-k-mooney | stephenfin: mriedem: commented on https://review.openstack.org/#/c/623564/7. | |
| 18:10:10 | mriedem | thanks | |
| 18:10:22 | mriedem | zzzeek: are you ok with https://review.openstack.org/#/c/623564/ now? | |
| 18:12:22 | sean-k-mooney | stephenfin: looking. i assuem that with both applied the guest with explcit small pages now works? | |
| 18:13:07 | stephenfin | sean-k-mooney: Yeah, the second test case from paste.openstack.org/show/742644/ now does what I'd expect | |
| 18:13:10 | stephenfin | testing the first case now | |
| 18:13:20 | sean-k-mooney | ack | |
| 18:29:01 | sean-k-mooney | stephenfin: left a comment on https://review.openstack.org/#/c/631038/1 | |
| 18:30:27 | cdent | thanks for the review on the gabbi-tempest stuff jaypipes . your comment is exactly why I like it too | |
| 18:31:19 | sean-k-mooney | cdent: gabbi-tempest test for placemet? | |
| 18:31:25 | melwitt | mriedem: thanks for the comments on the potential to leak host-related info here https://review.openstack.org/#/c/624832/5/nova/compute/manager.py I think the approach similar to InvalidBDM sounds safer, so I'm gonna respin | |
| 18:31:48 | cdent | sean-k-mooney: for nova: https://review.openstack.org/#/c/613386/6 | |
| 18:33:37 | jaypipes | cdent: I'm giddy to get me some good ol' gabbi. | |
| 18:34:15 | sean-k-mooney | ah i see. can gabbi also be used to simulatate an api? e.g. can i write the gabbits locally without needing and openstack/tempest deployment | |
| 18:34:15 | cdent | you've installed your alliterative poet guise today | |
| 18:34:37 | melwitt | cdent: I was about the say the same thing. alliterations and rhymes today I see | |
| 18:35:31 | cdent | sean-k-mooney: something needs to host the api. it can either be "live" like that tempest stuff, or with wsgi-intercept, as the placement functional tests do it | |
| 18:36:15 | sean-k-mooney | cdent: ah yes that was basically my intendt e.g. could you test it with say a tox job or something | |
| 18:36:20 | sean-k-mooney | *env | |
| 18:36:35 | sean-k-mooney | but ya that seams cool | |