| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-06 | |||
| 11:49:54 | openstackgerrit | Merged openstack/nova master: Remove old policy enforcement in attach_interfaces https://review.opendev.org/705127 | |
| 11:49:59 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing attach_interfaces policies https://review.opendev.org/705126 | |
| 11:50:07 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-attach-interfaces https://review.opendev.org/705799 | |
| 11:50:16 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-console-auth-tokens https://review.opendev.org/706688 | |
| 11:54:01 | happyhemant | stephenfin: hey hi stephenfin any idea about spoo checking caus it suppose to be ON. https://www.irccloud.com/pastebin/FHzoKqNc/ | |
| 11:55:31 | happyhemant | stephenfin: i tried to enable it but this is what am i getting on computes https://www.irccloud.com/pastebin/weWf4L2z/ | |
| 11:55:38 | happyhemant | what could be wrong here ? | |
| 11:57:07 | gibi | brinzhang: sorry, I did not see what is wrong with that functional test. I hope johnthetubaguy can help | |
| 12:28:00 | brinzhang | gibi: yeah, thanks. I think johnthethubaguy and gmann can help, they may need to talk something of that failed, that when I will join. | |
| 12:44:41 | openstackgerrit | Merged openstack/nova master: Correct the actual target in os-instance-actions policy https://review.opendev.org/710411 | |
| 12:44:47 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| 12:44:54 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-admin-password https://review.opendev.org/701630 | |
| 12:45:01 | openstackgerrit | Merged openstack/nova master: Add new default roles in os-admin-password policies https://review.opendev.org/701639 | |
| 12:45:08 | openstackgerrit | Merged openstack/nova master: Add a tests to check when legacy access is removed https://review.opendev.org/710813 | |
| 12:45:24 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing console_output policies https://review.opendev.org/706724 | |
| 12:53:41 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751 | |
| 12:55:22 | brinzhang | stephenfin: can you +A again, I was reabseed because of the conflict https://review.opendev.org/#/c/707751/ | |
| 12:56:23 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | |
| 12:56:36 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API https://review.opendev.org/706179 | |
| 13:13:39 | gibi | stephenfin: I'm +2 all the way up in the nova-net patches | |
| 13:29:37 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-console-output https://review.opendev.org/707040 | |
| 13:32:10 | openstackgerrit | Merged openstack/nova master: Add new default roles in os-console-output policies https://review.opendev.org/707041 | |
| 13:32:15 | openstackgerrit | Merged openstack/nova master: Fix os-volumes-attachments policy to be admin_or_owner https://review.opendev.org/709955 | |
| 13:32:24 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929 | |
| 13:36:38 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388 | |
| 13:36:51 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-aggregates policy https://review.opendev.org/701652 | |
| 13:48:57 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM zuul: Attempt to migrate and break up the nova-live-migration job https://review.opendev.org/711604 | |
| 13:53:54 | gmann | brinzhang: getting coffee. I will check the error. is it instance action one ? | |
| 14:02:46 | brinzhang | gmann: yes, it's the failed patch https://review.opendev.org/#/c/706470/ | |
| 14:05:00 | brinzhang | gmann: the failed is the non-admin expected response is not match the actual response, and the 'traceback' in the actual response, it's should return when the user is non-admin | |
| 14:05:22 | brinzhang | gmann: thanks for your check. | |
| 14:06:27 | gmann | ok | |
| 14:06:42 | brinzhang | johnthetubaguy: thanks +A for that rebase patch | |
| 14:10:22 | kashyap | bauzas: stephenfin: Heya, for allocating huge pages, have we already considered the existing libvirt API allocPages()? (I don't see it in the source from a lazy `grep`) | |
| 14:10:26 | kashyap | Documentation: https://libvirt.org/html/libvirt-libvirt-host.html#virNodeAllocPages | |
| 14:13:13 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug 1862633 https://review.opendev.org/711626 | |
| 14:13:13 | openstack | bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:16:41 | bauzas | kashyap: good question, I dunno | |
| 14:17:56 | kashyap | bauzas: Okay, I'll make a mental note to explore it. | |
| 14:22:33 | stephenfin | kashyap: What's the context? | |
| 14:22:57 | kashyap | stephenfin: I was triaging a downstream bug, where Nova ends up scheduling a VM to a host w/ insufficient huge pages | |
| 14:23:10 | kashyap | Therefore it fails to start on the dest host with: | |
| 14:23:31 | kashyap | [quote] | |
| 14:23:32 | kashyap | qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/22-instance-00000688,share=yes,size=17179869184,host-nodes=0,policy=bind: unable to map backing store for guest RAM: Cannot allocate memory | |
| 14:23:36 | kashyap | [/quote] | |
| 14:23:57 | stephenfin | kashyap: My guess is that they have something else on the host using hugepages and have configured the reserved hugepages config opt | |
| 14:24:42 | kashyap | stephenfin: So maybe we should get the huge pages usage from the host go from there | |
| 14:24:50 | kashyap | (And also the Nova config attribute you mention) | |
| 14:24:53 | kashyap | Thanks for the idea | |
| 14:24:59 | stephenfin | kashyap: '[DEFAULT] reserved_huge_pages', btw | |
| 14:25:12 | johnthetubaguy | I have seen that when the NUMA config was bad, by accident | |
| 14:25:37 | johnthetubaguy | i.e. not enough huge pages on a single numa node, as that is what I implicitly requested | |
| 14:26:03 | stephenfin | johnthetubaguy: But we do track hugepages on a per-node basis, right? | |
| 14:26:40 | stephenfin | so there must be something on that node outside of nova consuming those hugepages | |
| 14:26:50 | stephenfin | or our tracking is broken | |
| 14:27:11 | johnthetubaguy | I guess, I thought I saw that for that case too, but I kinda stopped digging for other reasons | |
| 14:27:21 | kashyap | So, Nova does something bespoke to account for memory? (/me should read the code) | |
| 14:27:21 | johnthetubaguy | it was a while back mind | |
| 14:27:58 | kashyap | johnthetubaguy: What do you mean "bad NUMA config"? What is a "good one", in Nova's parlance? | |
| 14:28:10 | stephenfin | kashyap: Look for references to NUMAPagesTopology | |
| 14:28:17 | stephenfin | we use that object to do our tracking | |
| 14:28:33 | johnthetubaguy | its more that if you don't specify it, you are requesting a single numa node with all the RAM | |
| 14:28:58 | johnthetubaguy | if you are trying to request the whole node, which is two numa nodes, that is going to fail | |
| 14:29:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629 | |
| 14:29:39 | kashyap | stephenfin: Noted; thanks for the pointer. | |
| 14:29:40 | stephenfin | so 64GB RAM from a two node host with exactly 64GB total (32GB per node)? | |
| 14:30:03 | johnthetubaguy | yeah, something like that | |
| 14:30:05 | stephenfin | gibi++ Thanks, btw :) | |
| 14:30:30 | stephenfin | brinzhang: looks like johnthetubaguy beat me to it | |
| 14:31:37 | johnthetubaguy | stephenfin: gmann has added some nice extra policy testing for you to take a peak at btw: https://review.opendev.org/#/c/707039/4/nova/tests/unit/policies/test_create_backup.py | |
| 14:32:00 | johnthetubaguy | basically test what happen after we lift all the deprecations, at some point in the future (or via deployer config) | |
| 14:33:07 | kashyap | stephenfin: Also, isn't it tricky (read: "needs bespoke script") to get the all the mapped huge pages usage by different applications on the host? | |
| 14:33:18 | gmann | brinzhang: johnthetubaguy seems we passed extra bit of project id here - https://review.opendev.org/#/c/710411/4/nova/api/openstack/compute/instance_actions.py@166 | |
| 14:33:34 | gmann | that rule was admin only and does not need project_id | |
| 14:34:07 | johnthetubaguy | gmann: good catch, should by empty dictionary there | |
| 14:35:19 | johnthetubaguy | gmann: mind you, the correct project is better than not specifying a target, so still a step forward I guess ;) | |
| 14:36:04 | kashyap | stephenfin: E.g. this short Perl script, which does some huge pages accounting: https://serverfault.com/questions/527085/linux-non-transparent-per-process-hugepage-accounting/644471#644471 | |
| 14:36:11 | gmann | johnthetubaguy: but in that case it can be used as owner also with override in rule | |
| 14:36:22 | johnthetubaguy | gmann: ... thinking about that more, if someone wants to make that non-admin (similar issue for live-migration) we will need that back in | |
| 14:36:34 | johnthetubaguy | gmann: heh, +1 what you just said, basically | |
| 14:37:15 | johnthetubaguy | gmann: this stuff is rock hard | |
| 14:37:20 | gmann | johnthetubaguy: yeah but it is right or wrong :) ? because we do not allow the same for any other admin only policy | |
| 14:37:28 | gmann | admin only by default | |
| 14:38:35 | johnthetubaguy | we probably should allow that, for things where it is not strictly a system level thing | |
| 14:39:04 | johnthetubaguy | but, we are getting ahead of ourselves I guess | |
| 14:39:48 | johnthetubaguy | this is about digging out of crazy land, so we can add more options in the future | |
| 14:40:01 | brinzhang | gmann, johnthetubaguy: I change that to the target={}, but it has the same issue, http://paste.openstack.org/show/790395/ | |
| 14:41:07 | johnthetubaguy | gmann: is brinzhang hitting the same issue you are here: https://review.opendev.org/#/c/706689 | |
| 14:41:16 | johnthetubaguy | I am not sure what is causing the problem though :( | |
| 14:41:24 | gmann | johnthetubaguy: ok, let's go case by case and judge the not-strict-system-cases or upon user request . i will leave instance action also in that catagory | |
| 14:41:36 | johnthetubaguy | gmann: +1 | |
| 14:42:14 | gmann | johnthetubaguy: brinzhang i think i observed some wired things on aggregate policy also but could not debug those yet. let me debug it | |
| 14:42:31 | johnthetubaguy | gmann: +1 thanks | |
| 14:42:53 | brinzhang | gmann: thanks | |
| 14:43:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug 1862633 https://review.opendev.org/711626 | |
| 14:43:50 | openstack | bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:44:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629 | |
| 14:46:37 | gibi | stephenfin: would be nice to get a second opinion on https://review.opendev.org/#/c/666245/ It seems to be working but I'm a bit affraid of what this change means on a busy deployment | |
| 14:55:50 | stephenfin | gibi: ack, will do | |
| 14:56:32 | mloza | I specificied cpu_models=x86_EPYC-IBRS in nova.conf but I keep getting this error ': Invalid: Config requested a custom CPU model, but no model name was provided' | |