| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-06 | |||
| 10:32:27 | johnthetubaguy | brinzhang_: try adding the system_scope=all here: https://github.com/openstack/nova/blob/f176ffe6e57e21c4c5d9f33a3c9e47476d10acc9/nova/tests/fixtures.py#L916 | |
| 10:32:47 | johnthetubaguy | not totally sure... but I think that is the place | |
| 10:32:49 | brinzhang_ | johnthetubaguy: ok, let me try | |
| 10:38:05 | brinzhang_ | johnthetubaguy: do you point here https://opendev.org/openstack/nova/src/branch/master/nova/tests/fixtures.py#L916? we can not open github fastly | |
| 10:38:47 | gibi | stephenfin: I will try to look at those today | |
| 10:39:09 | johnthetubaguy | brinzhang_: yeah, sorry, that is where I meant, it is created a fake object, line 948 is what that code is matching on | |
| 10:39:13 | stephenfin | Thanks. Let me know if anything isn't immediately obvious :) | |
| 10:39:14 | brinzhang_ | johnthetubaguy: That changed to "is_admin = user_id == 'admin and system_scope=all'"? | |
| 10:39:39 | stephenfin | gibi: Also, all hail the new leader 👑 | |
| 10:39:39 | stephenfin | gibi: Also, all hail the new leader 👑 | |
| 10:39:58 | johnthetubaguy | ah, no, I think you need to add system_scope='all' into the kwargs when user_id == 'admin' | |
| 10:40:17 | johnthetubaguy | basically, so it looks like we get a new style admin | |
| 10:40:45 | brinzhang_ | johnthetubaguy: sure | |
| 10:41:06 | johnthetubaguy | ... having said that, it does suggest the deprecation rules are not quite working as expected, I would ask gmann for more info | |
| 10:43:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs https://review.opendev.org/706295 | |
| 10:47:03 | brinzhang_ | johnthetubaguy: it seems not correct, the failed test case is fot non-admin, if we add system_scope='admin' if is_admin=True, that we just change the admin's context, the non-admin context also same as before | |
| 10:47:12 | brinzhang_ | and it has the same issue | |
| 10:47:44 | johnthetubaguy | ah, OK, I missed that | |
| 10:47:59 | brinzhang_ | http://paste.openstack.org/show/790390/ this is my change, and the test result alos same before | |
| 10:47:59 | johnthetubaguy | curious | |
| 10:48:26 | brinzhang_ | and I think the if always False. | |
| 11:05:26 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM zuul: Attempt to migrate and break up the nova-live-migration job https://review.opendev.org/711604 | |
| 11:37:21 | openstackgerrit | Merged openstack/nova stable/train: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/709167 | |
| 11:37:43 | openstackgerrit | Qiu Fossen proposed openstack/nova master: The instance is volume backed and power state is PAUSED，shelve the instance failed https://review.opendev.org/711609 | |
| 11:37:43 | openstackgerrit | Qiu Fossen proposed openstack/nova master: The instance is volume backed and power state is PAUSED，shelve the instance failed https://review.opendev.org/711609 | |
| 11:46:43 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates https://review.opendev.org/710214 | |
| 11:49:54 | openstackgerrit | Merged openstack/nova master: Remove old policy enforcement in attach_interfaces https://review.opendev.org/705127 | |
| 11:49:59 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing attach_interfaces policies https://review.opendev.org/705126 | |
| 11:50:07 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-attach-interfaces https://review.opendev.org/705799 | |
| 11:50:16 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-console-auth-tokens https://review.opendev.org/706688 | |
| 11:54:01 | happyhemant | stephenfin: hey hi stephenfin any idea about spoo checking caus it suppose to be ON. https://www.irccloud.com/pastebin/FHzoKqNc/ | |
| 11:55:31 | happyhemant | stephenfin: i tried to enable it but this is what am i getting on computes https://www.irccloud.com/pastebin/weWf4L2z/ | |
| 11:55:38 | happyhemant | what could be wrong here ? | |
| 11:57:07 | gibi | brinzhang: sorry, I did not see what is wrong with that functional test. I hope johnthetubaguy can help | |
| 12:28:00 | brinzhang | gibi: yeah, thanks. I think johnthethubaguy and gmann can help, they may need to talk something of that failed, that when I will join. | |
| 12:44:41 | openstackgerrit | Merged openstack/nova master: Correct the actual target in os-instance-actions policy https://review.opendev.org/710411 | |
| 12:44:47 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| 12:44:54 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-admin-password https://review.opendev.org/701630 | |
| 12:45:01 | openstackgerrit | Merged openstack/nova master: Add new default roles in os-admin-password policies https://review.opendev.org/701639 | |
| 12:45:08 | openstackgerrit | Merged openstack/nova master: Add a tests to check when legacy access is removed https://review.opendev.org/710813 | |
| 12:45:24 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing console_output policies https://review.opendev.org/706724 | |
| 12:53:41 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751 | |
| 12:55:22 | brinzhang | stephenfin: can you +A again, I was reabseed because of the conflict https://review.opendev.org/#/c/707751/ | |
| 12:56:23 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | |
| 12:56:36 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API https://review.opendev.org/706179 | |
| 13:13:39 | gibi | stephenfin: I'm +2 all the way up in the nova-net patches | |
| 13:29:37 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-console-output https://review.opendev.org/707040 | |
| 13:32:10 | openstackgerrit | Merged openstack/nova master: Add new default roles in os-console-output policies https://review.opendev.org/707041 | |
| 13:32:15 | openstackgerrit | Merged openstack/nova master: Fix os-volumes-attachments policy to be admin_or_owner https://review.opendev.org/709955 | |
| 13:32:24 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929 | |
| 13:36:38 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388 | |
| 13:36:51 | openstackgerrit | Merged openstack/nova master: Introduce scope_types in os-aggregates policy https://review.opendev.org/701652 | |
| 13:48:57 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM zuul: Attempt to migrate and break up the nova-live-migration job https://review.opendev.org/711604 | |
| 13:53:54 | gmann | brinzhang: getting coffee. I will check the error. is it instance action one ? | |
| 14:02:46 | brinzhang | gmann: yes, it's the failed patch https://review.opendev.org/#/c/706470/ | |
| 14:05:00 | brinzhang | gmann: the failed is the non-admin expected response is not match the actual response, and the 'traceback' in the actual response, it's should return when the user is non-admin | |
| 14:05:22 | brinzhang | gmann: thanks for your check. | |
| 14:06:27 | gmann | ok | |
| 14:06:42 | brinzhang | johnthetubaguy: thanks +A for that rebase patch | |
| 14:10:22 | kashyap | bauzas: stephenfin: Heya, for allocating huge pages, have we already considered the existing libvirt API allocPages()? (I don't see it in the source from a lazy `grep`) | |
| 14:10:26 | kashyap | Documentation: https://libvirt.org/html/libvirt-libvirt-host.html#virNodeAllocPages | |
| 14:13:13 | openstack | bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:13:13 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug 1862633 https://review.opendev.org/711626 | |
| 14:16:41 | bauzas | kashyap: good question, I dunno | |
| 14:17:56 | kashyap | bauzas: Okay, I'll make a mental note to explore it. | |
| 14:22:33 | stephenfin | kashyap: What's the context? | |
| 14:22:57 | kashyap | stephenfin: I was triaging a downstream bug, where Nova ends up scheduling a VM to a host w/ insufficient huge pages | |
| 14:23:10 | kashyap | Therefore it fails to start on the dest host with: | |
| 14:23:31 | kashyap | [quote] | |
| 14:23:32 | kashyap | qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/22-instance-00000688,share=yes,size=17179869184,host-nodes=0,policy=bind: unable to map backing store for guest RAM: Cannot allocate memory | |
| 14:23:36 | kashyap | [/quote] | |
| 14:23:57 | stephenfin | kashyap: My guess is that they have something else on the host using hugepages and have configured the reserved hugepages config opt | |
| 14:24:42 | kashyap | stephenfin: So maybe we should get the huge pages usage from the host go from there | |
| 14:24:50 | kashyap | (And also the Nova config attribute you mention) | |
| 14:24:53 | kashyap | Thanks for the idea | |
| 14:24:59 | stephenfin | kashyap: '[DEFAULT] reserved_huge_pages', btw | |
| 14:25:12 | johnthetubaguy | I have seen that when the NUMA config was bad, by accident | |
| 14:25:37 | johnthetubaguy | i.e. not enough huge pages on a single numa node, as that is what I implicitly requested | |
| 14:26:03 | stephenfin | johnthetubaguy: But we do track hugepages on a per-node basis, right? | |
| 14:26:40 | stephenfin | so there must be something on that node outside of nova consuming those hugepages | |
| 14:26:50 | stephenfin | or our tracking is broken | |
| 14:27:11 | johnthetubaguy | I guess, I thought I saw that for that case too, but I kinda stopped digging for other reasons | |
| 14:27:21 | johnthetubaguy | it was a while back mind | |
| 14:27:21 | kashyap | So, Nova does something bespoke to account for memory? (/me should read the code) | |
| 14:27:58 | kashyap | johnthetubaguy: What do you mean "bad NUMA config"? What is a "good one", in Nova's parlance? | |
| 14:28:10 | stephenfin | kashyap: Look for references to NUMAPagesTopology | |
| 14:28:17 | stephenfin | we use that object to do our tracking | |
| 14:28:33 | johnthetubaguy | its more that if you don't specify it, you are requesting a single numa node with all the RAM | |
| 14:28:58 | johnthetubaguy | if you are trying to request the whole node, which is two numa nodes, that is going to fail | |
| 14:29:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629 | |
| 14:29:39 | kashyap | stephenfin: Noted; thanks for the pointer. | |
| 14:29:40 | stephenfin | so 64GB RAM from a two node host with exactly 64GB total (32GB per node)? | |
| 14:30:03 | johnthetubaguy | yeah, something like that | |
| 14:30:05 | stephenfin | gibi++ Thanks, btw :) | |
| 14:30:30 | stephenfin | brinzhang: looks like johnthetubaguy beat me to it | |
| 14:31:37 | johnthetubaguy | stephenfin: gmann has added some nice extra policy testing for you to take a peak at btw: https://review.opendev.org/#/c/707039/4/nova/tests/unit/policies/test_create_backup.py | |
| 14:32:00 | johnthetubaguy | basically test what happen after we lift all the deprecations, at some point in the future (or via deployer config) | |
| 14:33:07 | kashyap | stephenfin: Also, isn't it tricky (read: "needs bespoke script") to get the all the mapped huge pages usage by different applications on the host? | |
| 14:33:18 | gmann | brinzhang: johnthetubaguy seems we passed extra bit of project id here - https://review.opendev.org/#/c/710411/4/nova/api/openstack/compute/instance_actions.py@166 | |
| 14:33:34 | gmann | that rule was admin only and does not need project_id | |