Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-06
14:25:37 johnthetubaguy i.e. not enough huge pages on a single numa node, as that is what I implicitly requested
14:26:03 stephenfin johnthetubaguy: But we do track hugepages on a per-node basis, right?
14:26:40 stephenfin so there must be something on that node outside of nova consuming those hugepages
14:26:50 stephenfin or our tracking is broken
14:27:11 johnthetubaguy I guess, I thought I saw that for that case too, but I kinda stopped digging for other reasons
14:27:21 johnthetubaguy it was a while back mind
14:27:21 kashyap So, Nova does something bespoke to account for memory? (/me should read the code)
14:27:58 kashyap johnthetubaguy: What do you mean "bad NUMA config"? What is a "good one", in Nova's parlance?
14:28:10 stephenfin kashyap: Look for references to NUMAPagesTopology
14:28:17 stephenfin we use that object to do our tracking
14:28:33 johnthetubaguy its more that if you don't specify it, you are requesting a single numa node with all the RAM
14:28:58 johnthetubaguy if you are trying to request the whole node, which is two numa nodes, that is going to fail
14:29:30 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629
14:29:39 kashyap stephenfin: Noted; thanks for the pointer.
14:29:40 stephenfin so 64GB RAM from a two node host with exactly 64GB total (32GB per node)?
14:30:03 johnthetubaguy yeah, something like that
14:30:05 stephenfin gibi++ Thanks, btw :)
14:30:30 stephenfin brinzhang: looks like johnthetubaguy beat me to it
14:31:37 johnthetubaguy stephenfin: gmann has added some nice extra policy testing for you to take a peak at btw: https://review.opendev.org/#/c/707039/4/nova/tests/unit/policies/test_create_backup.py
14:32:00 johnthetubaguy basically test what happen after we lift all the deprecations, at some point in the future (or via deployer config)
14:33:07 kashyap stephenfin: Also, isn't it tricky (read: "needs bespoke script") to get the all the mapped huge pages usage by different applications on the host?
14:33:18 gmann brinzhang: johnthetubaguy seems we passed extra bit of project id here - https://review.opendev.org/#/c/710411/4/nova/api/openstack/compute/instance_actions.py@166
14:33:34 gmann that rule was admin only and does not need project_id
14:34:07 johnthetubaguy gmann: good catch, should by empty dictionary there
14:35:19 johnthetubaguy gmann: mind you, the correct project is better than not specifying a target, so still a step forward I guess ;)
14:36:04 kashyap stephenfin: E.g. this short Perl script, which does some huge pages accounting: https://serverfault.com/questions/527085/linux-non-transparent-per-process-hugepage-accounting/644471#644471
14:36:11 gmann johnthetubaguy: but in that case it can be used as owner also with override in rule
14:36:22 johnthetubaguy gmann: ... thinking about that more, if someone wants to make that non-admin (similar issue for live-migration) we will need that back in
14:36:34 johnthetubaguy gmann: heh, +1 what you just said, basically
14:37:15 johnthetubaguy gmann: this stuff is rock hard
14:37:20 gmann johnthetubaguy: yeah but it is right or wrong :) ? because we do not allow the same for any other admin only policy
14:37:28 gmann admin only by default
14:38:35 johnthetubaguy we probably should allow that, for things where it is not strictly a system level thing
14:39:04 johnthetubaguy but, we are getting ahead of ourselves I guess
14:39:48 johnthetubaguy this is about digging out of crazy land, so we can add more options in the future
14:40:01 brinzhang gmann, johnthetubaguy: I change that to the target={}, but it has the same issue, http://paste.openstack.org/show/790395/
14:41:07 johnthetubaguy gmann: is brinzhang hitting the same issue you are here: https://review.opendev.org/#/c/706689
14:41:16 johnthetubaguy I am not sure what is causing the problem though :(
14:41:24 gmann johnthetubaguy: ok, let's go case by case and judge the not-strict-system-cases or upon user request . i will leave instance action also in that catagory
14:41:36 johnthetubaguy gmann: +1
14:42:14 gmann johnthetubaguy: brinzhang i think i observed some wired things on aggregate policy also but could not debug those yet. let me debug it
14:42:31 johnthetubaguy gmann: +1 thanks
14:42:53 brinzhang gmann: thanks
14:43:50 openstack bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer)
14:43:50 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug 1862633 https://review.opendev.org/711626
14:44:08 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629
14:46:37 gibi stephenfin: would be nice to get a second opinion on https://review.opendev.org/#/c/666245/ It seems to be working but I'm a bit affraid of what this change means on a busy deployment
14:55:50 stephenfin gibi: ack, will do
14:56:32 mloza I specificied cpu_models=x86_EPYC-IBRS in nova.conf but I keep getting this error ': Invalid: Config requested a custom CPU model, but no model name was provided'
14:56:54 mloza The model exist in /usr/share/libvirt/cpu_map/x86_EPYC-IBRS.xml
14:57:07 mloza This is in stable/stein
14:59:18 openstackgerrit Lee Yarwood proposed openstack/nova master: workarounds: Add option to disable native LUKSv1 decryption by QEMU https://review.opendev.org/708030
14:59:19 openstackgerrit Lee Yarwood proposed openstack/nova master: workarounds: Connect RBD volumes to the compute host as block devices https://review.opendev.org/708029
15:28:36 johnthetubaguy mloza: I think stein only supports the option cpu_model: https://docs.openstack.org/nova/stein/configuration/config.html#libvirt.cpu_model
15:33:12 openstackgerrit Stephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs https://review.opendev.org/706295
15:37:13 mloza johnthetubaguy: yeah. I just noticed now. I was checking the latest docs of nova. I didn't know that they renamed cpu_model to cpu_models in the train release
15:47:28 openstackgerrit Merged openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751
15:56:36 stephenfin gibi: so what were your specific concerns for https://review.opendev.org/#/c/666245/ ?
16:15:40 stephenfin dansmith: Can this go ahead now? https://review.opendev.org/#/c/708707/ I'm guessing it's blocking the alembic upper-constraint bump (https://review.opendev.org/#/c/708706/)
16:15:58 dansmith I was just looking since I got the review email from you
16:16:11 dansmith I think this had to be tested against the bump,
16:16:17 dansmith and then again against the current version
16:16:21 dansmith so since it's passing now I guess?
16:16:37 dansmith prometheanfire: ^
16:27:22 prometheanfire we talking about the alembic thing?
16:27:33 prometheanfire I'm waiting on nova, so no blocker from me
16:28:00 dansmith prometheanfire: https://review.opendev.org/#/c/708707/4
16:28:43 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP/DNM zuul: Attempt to migrate and break up the nova-live-migration job https://review.opendev.org/711604
16:30:13 gibi stephenfin: that patch introduces a waitall call for the nova-compute stop codepath. So I can imagine it can change the timing of the nova-compute service restart scenarios in real deployments
16:31:27 gibi stephenfin: e.g. nova-compute will wait for the all the ongoing instances to boot up before it stops which could take significant time due to e.g. image download
16:31:27 stephenfin gibi: Yup. There will definitely be knock on effects. That's only for graceful shutdown though, so people could still force things, right?
16:31:44 gibi right, tooling can send kill signalls for sure
16:32:17 dansmith also probably long-running periodics, like resource update for lots of ironic nodes
16:32:41 stephenfin It's not unlimited though. Not by default
16:32:56 stephenfin I checked and graceful_shutdown_timeout defaults to a minute
16:32:58 dansmith no, it's not
16:33:16 dansmith actually, nevermind what I said about the resource update, because that does drop and re-acquire
16:33:19 stephenfin so at worst we're introducing an extra 60 second delay
16:33:36 dansmith stephenfin: where's the 60 second limit?
16:34:02 stephenfin dansmith: https://github.com/openstack/oslo.service/blob/master/oslo_service/_options.py#L51
16:34:06 dansmith is that something the service does?
16:34:26 stephenfin yeah, if this comment is to be believed https://review.opendev.org/#/c/666245/16/nova/compute/manager.py@1511
16:34:28 dansmith where's the implementation of that though?
16:34:50 dansmith https://github.com/openstack/oslo.service/blob/master/oslo_service/service.py#L333
16:35:21 dansmith does it with sigalrm so should break us out of that otherwise infinite wait
16:37:25 gibi OK, so it is really just an extra 60 seconds delay
16:37:37 stephenfin that's my understanding, yes
16:37:50 stephenfin I think that's reasonable. We have a reno too
16:38:15 gibi I feel better that none of you freaked out about that patch.
16:38:23 dansmith I haven't really reviewed
16:38:27 dansmith can I have a few minutes?
16:38:32 gibi dansmith: sure
16:38:47 gibi dansmith: but you already know the basic idea of the patch
16:38:56 dansmith yeah
16:39:05 dansmith we already have a build semaphore and config for it
16:39:14 dansmith this would seem to complicate that a bit
16:40:06 stephenfin Does this need a microversion? https://review.opendev.org/#/c/709431/
16:40:46 stephenfin tl;dr: We've been using '/hypervisors' in our 'links' pagination field, rather than '/os-hypervisors'
16:41:11 gibi stephenfin: I think we don't want API users to opt into a bugfix
16:41:30 stephenfin yes, that's my thinking too

Earlier   Later