| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-06 | |||
| 14:05:00 | brinzhang | gmann: the failed is the non-admin expected response is not match the actual response, and the 'traceback' in the actual response, it's should return when the user is non-admin | |
| 14:05:22 | brinzhang | gmann: thanks for your check. | |
| 14:06:27 | gmann | ok | |
| 14:06:42 | brinzhang | johnthetubaguy: thanks +A for that rebase patch | |
| 14:10:22 | kashyap | bauzas: stephenfin: Heya, for allocating huge pages, have we already considered the existing libvirt API allocPages()? (I don't see it in the source from a lazy `grep`) | |
| 14:10:26 | kashyap | Documentation: https://libvirt.org/html/libvirt-libvirt-host.html#virNodeAllocPages | |
| 14:13:13 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug 1862633 https://review.opendev.org/711626 | |
| 14:13:13 | openstack | bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:16:41 | bauzas | kashyap: good question, I dunno | |
| 14:17:56 | kashyap | bauzas: Okay, I'll make a mental note to explore it. | |
| 14:22:33 | stephenfin | kashyap: What's the context? | |
| 14:22:57 | kashyap | stephenfin: I was triaging a downstream bug, where Nova ends up scheduling a VM to a host w/ insufficient huge pages | |
| 14:23:10 | kashyap | Therefore it fails to start on the dest host with: | |
| 14:23:31 | kashyap | [quote] | |
| 14:23:32 | kashyap | qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/22-instance-00000688,share=yes,size=17179869184,host-nodes=0,policy=bind: unable to map backing store for guest RAM: Cannot allocate memory | |
| 14:23:36 | kashyap | [/quote] | |
| 14:23:57 | stephenfin | kashyap: My guess is that they have something else on the host using hugepages and have configured the reserved hugepages config opt | |
| 14:24:42 | kashyap | stephenfin: So maybe we should get the huge pages usage from the host go from there | |
| 14:24:50 | kashyap | (And also the Nova config attribute you mention) | |
| 14:24:53 | kashyap | Thanks for the idea | |
| 14:24:59 | stephenfin | kashyap: '[DEFAULT] reserved_huge_pages', btw | |
| 14:25:12 | johnthetubaguy | I have seen that when the NUMA config was bad, by accident | |
| 14:25:37 | johnthetubaguy | i.e. not enough huge pages on a single numa node, as that is what I implicitly requested | |
| 14:26:03 | stephenfin | johnthetubaguy: But we do track hugepages on a per-node basis, right? | |
| 14:26:40 | stephenfin | so there must be something on that node outside of nova consuming those hugepages | |
| 14:26:50 | stephenfin | or our tracking is broken | |
| 14:27:11 | johnthetubaguy | I guess, I thought I saw that for that case too, but I kinda stopped digging for other reasons | |
| 14:27:21 | kashyap | So, Nova does something bespoke to account for memory? (/me should read the code) | |
| 14:27:21 | johnthetubaguy | it was a while back mind | |
| 14:27:58 | kashyap | johnthetubaguy: What do you mean "bad NUMA config"? What is a "good one", in Nova's parlance? | |
| 14:28:10 | stephenfin | kashyap: Look for references to NUMAPagesTopology | |
| 14:28:17 | stephenfin | we use that object to do our tracking | |
| 14:28:33 | johnthetubaguy | its more that if you don't specify it, you are requesting a single numa node with all the RAM | |
| 14:28:58 | johnthetubaguy | if you are trying to request the whole node, which is two numa nodes, that is going to fail | |
| 14:29:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629 | |
| 14:29:39 | kashyap | stephenfin: Noted; thanks for the pointer. | |
| 14:29:40 | stephenfin | so 64GB RAM from a two node host with exactly 64GB total (32GB per node)? | |
| 14:30:03 | johnthetubaguy | yeah, something like that | |
| 14:30:05 | stephenfin | gibi++ Thanks, btw :) | |
| 14:30:30 | stephenfin | brinzhang: looks like johnthetubaguy beat me to it | |
| 14:31:37 | johnthetubaguy | stephenfin: gmann has added some nice extra policy testing for you to take a peak at btw: https://review.opendev.org/#/c/707039/4/nova/tests/unit/policies/test_create_backup.py | |
| 14:32:00 | johnthetubaguy | basically test what happen after we lift all the deprecations, at some point in the future (or via deployer config) | |
| 14:33:07 | kashyap | stephenfin: Also, isn't it tricky (read: "needs bespoke script") to get the all the mapped huge pages usage by different applications on the host? | |
| 14:33:18 | gmann | brinzhang: johnthetubaguy seems we passed extra bit of project id here - https://review.opendev.org/#/c/710411/4/nova/api/openstack/compute/instance_actions.py@166 | |
| 14:33:34 | gmann | that rule was admin only and does not need project_id | |
| 14:34:07 | johnthetubaguy | gmann: good catch, should by empty dictionary there | |
| 14:35:19 | johnthetubaguy | gmann: mind you, the correct project is better than not specifying a target, so still a step forward I guess ;) | |
| 14:36:04 | kashyap | stephenfin: E.g. this short Perl script, which does some huge pages accounting: https://serverfault.com/questions/527085/linux-non-transparent-per-process-hugepage-accounting/644471#644471 | |
| 14:36:11 | gmann | johnthetubaguy: but in that case it can be used as owner also with override in rule | |
| 14:36:22 | johnthetubaguy | gmann: ... thinking about that more, if someone wants to make that non-admin (similar issue for live-migration) we will need that back in | |
| 14:36:34 | johnthetubaguy | gmann: heh, +1 what you just said, basically | |
| 14:37:15 | johnthetubaguy | gmann: this stuff is rock hard | |
| 14:37:20 | gmann | johnthetubaguy: yeah but it is right or wrong :) ? because we do not allow the same for any other admin only policy | |
| 14:37:28 | gmann | admin only by default | |
| 14:38:35 | johnthetubaguy | we probably should allow that, for things where it is not strictly a system level thing | |
| 14:39:04 | johnthetubaguy | but, we are getting ahead of ourselves I guess | |
| 14:39:48 | johnthetubaguy | this is about digging out of crazy land, so we can add more options in the future | |
| 14:40:01 | brinzhang | gmann, johnthetubaguy: I change that to the target={}, but it has the same issue, http://paste.openstack.org/show/790395/ | |
| 14:41:07 | johnthetubaguy | gmann: is brinzhang hitting the same issue you are here: https://review.opendev.org/#/c/706689 | |
| 14:41:16 | johnthetubaguy | I am not sure what is causing the problem though :( | |
| 14:41:24 | gmann | johnthetubaguy: ok, let's go case by case and judge the not-strict-system-cases or upon user request . i will leave instance action also in that catagory | |
| 14:41:36 | johnthetubaguy | gmann: +1 | |
| 14:42:14 | gmann | johnthetubaguy: brinzhang i think i observed some wired things on aggregate policy also but could not debug those yet. let me debug it | |
| 14:42:31 | johnthetubaguy | gmann: +1 thanks | |
| 14:42:53 | brinzhang | gmann: thanks | |
| 14:43:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug 1862633 https://review.opendev.org/711626 | |
| 14:43:50 | openstack | bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:44:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629 | |
| 14:46:37 | gibi | stephenfin: would be nice to get a second opinion on https://review.opendev.org/#/c/666245/ It seems to be working but I'm a bit affraid of what this change means on a busy deployment | |
| 14:55:50 | stephenfin | gibi: ack, will do | |
| 14:56:32 | mloza | I specificied cpu_models=x86_EPYC-IBRS in nova.conf but I keep getting this error ': Invalid: Config requested a custom CPU model, but no model name was provided' | |
| 14:56:54 | mloza | The model exist in /usr/share/libvirt/cpu_map/x86_EPYC-IBRS.xml | |
| 14:57:07 | mloza | This is in stable/stein | |
| 14:59:18 | openstackgerrit | Lee Yarwood proposed openstack/nova master: workarounds: Add option to disable native LUKSv1 decryption by QEMU https://review.opendev.org/708030 | |
| 14:59:19 | openstackgerrit | Lee Yarwood proposed openstack/nova master: workarounds: Connect RBD volumes to the compute host as block devices https://review.opendev.org/708029 | |
| 15:28:36 | johnthetubaguy | mloza: I think stein only supports the option cpu_model: https://docs.openstack.org/nova/stein/configuration/config.html#libvirt.cpu_model | |
| 15:33:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs https://review.opendev.org/706295 | |
| 15:37:13 | mloza | johnthetubaguy: yeah. I just noticed now. I was checking the latest docs of nova. I didn't know that they renamed cpu_model to cpu_models in the train release | |
| 15:47:28 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751 | |
| 15:56:36 | stephenfin | gibi: so what were your specific concerns for https://review.opendev.org/#/c/666245/ ? | |
| 16:15:40 | stephenfin | dansmith: Can this go ahead now? https://review.opendev.org/#/c/708707/ I'm guessing it's blocking the alembic upper-constraint bump (https://review.opendev.org/#/c/708706/) | |
| 16:15:58 | dansmith | I was just looking since I got the review email from you | |
| 16:16:11 | dansmith | I think this had to be tested against the bump, | |
| 16:16:17 | dansmith | and then again against the current version | |
| 16:16:21 | dansmith | so since it's passing now I guess? | |
| 16:16:37 | dansmith | prometheanfire: ^ | |
| 16:27:22 | prometheanfire | we talking about the alembic thing? | |
| 16:27:33 | prometheanfire | I'm waiting on nova, so no blocker from me | |
| 16:28:00 | dansmith | prometheanfire: https://review.opendev.org/#/c/708707/4 | |
| 16:28:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM zuul: Attempt to migrate and break up the nova-live-migration job https://review.opendev.org/711604 | |
| 16:30:13 | gibi | stephenfin: that patch introduces a waitall call for the nova-compute stop codepath. So I can imagine it can change the timing of the nova-compute service restart scenarios in real deployments | |
| 16:31:27 | stephenfin | gibi: Yup. There will definitely be knock on effects. That's only for graceful shutdown though, so people could still force things, right? | |
| 16:31:27 | gibi | stephenfin: e.g. nova-compute will wait for the all the ongoing instances to boot up before it stops which could take significant time due to e.g. image download | |
| 16:31:44 | gibi | right, tooling can send kill signalls for sure | |
| 16:32:17 | dansmith | also probably long-running periodics, like resource update for lots of ironic nodes | |
| 16:32:41 | stephenfin | It's not unlimited though. Not by default | |
| 16:32:56 | stephenfin | I checked and graceful_shutdown_timeout defaults to a minute | |
| 16:32:58 | dansmith | no, it's not | |
| 16:33:16 | dansmith | actually, nevermind what I said about the resource update, because that does drop and re-acquire | |
| 16:33:19 | stephenfin | so at worst we're introducing an extra 60 second delay | |