Posted	Nick	Remark
#openstack-nova - 2020-03-06
10:32:47	johnthetubaguy	not totally sure... but I think that is the place
10:32:49	brinzhang_	johnthetubaguy: ok, let me try
10:38:05	brinzhang_	johnthetubaguy: do you point here https://opendev.org/openstack/nova/src/branch/master/nova/tests/fixtures.py#L916? we can not open github fastly
10:38:47	gibi	stephenfin: I will try to look at those today
10:39:09	johnthetubaguy	brinzhang_: yeah, sorry, that is where I meant, it is created a fake object, line 948 is what that code is matching on
10:39:13	stephenfin	Thanks. Let me know if anything isn't immediately obvious :)
10:39:14	brinzhang_	johnthetubaguy: That changed to "is_admin = user_id == 'admin and system_scope=all'"?
10:39:39	stephenfin	gibi: Also, all hail the new leader 👑
10:39:39	stephenfin	gibi: Also, all hail the new leader 👑
10:39:58	johnthetubaguy	ah, no, I think you need to add system_scope='all' into the kwargs when user_id == 'admin'
10:40:17	johnthetubaguy	basically, so it looks like we get a new style admin
10:40:45	brinzhang_	johnthetubaguy: sure
10:41:06	johnthetubaguy	... having said that, it does suggest the deprecation rules are not quite working as expected, I would ask gmann for more info
10:43:43	openstackgerrit	Stephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs https://review.opendev.org/706295
10:47:03	brinzhang_	johnthetubaguy: it seems not correct, the failed test case is fot non-admin, if we add system_scope='admin' if is_admin=True, that we just change the admin's context, the non-admin context also same as before
10:47:12	brinzhang_	and it has the same issue
10:47:44	johnthetubaguy	ah, OK, I missed that
10:47:59	brinzhang_	http://paste.openstack.org/show/790390/ this is my change, and the test result alos same before
10:47:59	johnthetubaguy	curious
10:48:26	brinzhang_	and I think the if always False.
11:05:26	openstackgerrit	Lee Yarwood proposed openstack/nova master: WIP/DNM zuul: Attempt to migrate and break up the nova-live-migration job https://review.opendev.org/711604
11:37:21	openstackgerrit	Merged openstack/nova stable/train: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/709167
11:37:43	openstackgerrit	Qiu Fossen proposed openstack/nova master: The instance is volume backed and power state is PAUSED，shelve the instance failed https://review.opendev.org/711609
11:37:43	openstackgerrit	Qiu Fossen proposed openstack/nova master: The instance is volume backed and power state is PAUSED，shelve the instance failed https://review.opendev.org/711609
11:46:43	openstackgerrit	Johannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates https://review.opendev.org/710214
11:49:54	openstackgerrit	Merged openstack/nova master: Remove old policy enforcement in attach_interfaces https://review.opendev.org/705127
11:49:59	openstackgerrit	Merged openstack/nova master: Add test coverage of existing attach_interfaces policies https://review.opendev.org/705126
11:50:07	openstackgerrit	Merged openstack/nova master: Introduce scope_types in os-attach-interfaces https://review.opendev.org/705799
11:50:16	openstackgerrit	Merged openstack/nova master: Introduce scope_types in os-console-auth-tokens https://review.opendev.org/706688
11:54:01	happyhemant	stephenfin: hey hi stephenfin any idea about spoo checking caus it suppose to be ON. https://www.irccloud.com/pastebin/FHzoKqNc/
11:55:31	happyhemant	stephenfin: i tried to enable it but this is what am i getting on computes https://www.irccloud.com/pastebin/weWf4L2z/
11:55:38	happyhemant	what could be wrong here ?
11:57:07	gibi	brinzhang: sorry, I did not see what is wrong with that functional test. I hope johnthetubaguy can help
12:28:00	brinzhang	gibi: yeah, thanks. I think johnthethubaguy and gmann can help, they may need to talk something of that failed, that when I will join.
12:44:41	openstackgerrit	Merged openstack/nova master: Correct the actual target in os-instance-actions policy https://review.opendev.org/710411
12:44:47	openstackgerrit	Merged openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777
12:44:54	openstackgerrit	Merged openstack/nova master: Introduce scope_types in os-admin-password https://review.opendev.org/701630
12:45:01	openstackgerrit	Merged openstack/nova master: Add new default roles in os-admin-password policies https://review.opendev.org/701639
12:45:08	openstackgerrit	Merged openstack/nova master: Add a tests to check when legacy access is removed https://review.opendev.org/710813
12:45:24	openstackgerrit	Merged openstack/nova master: Add test coverage of existing console_output policies https://review.opendev.org/706724
12:53:41	openstackgerrit	Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751
12:55:22	brinzhang	stephenfin: can you +A again, I was reabseed because of the conflict https://review.opendev.org/#/c/707751/
12:56:23	openstackgerrit	Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470
12:56:36	openstackgerrit	Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API https://review.opendev.org/706179
13:13:39	gibi	stephenfin: I'm +2 all the way up in the nova-net patches
13:29:37	openstackgerrit	Merged openstack/nova master: Introduce scope_types in os-console-output https://review.opendev.org/707040
13:32:10	openstackgerrit	Merged openstack/nova master: Add new default roles in os-console-output policies https://review.opendev.org/707041
13:32:15	openstackgerrit	Merged openstack/nova master: Fix os-volumes-attachments policy to be admin_or_owner https://review.opendev.org/709955
13:32:24	openstackgerrit	Merged openstack/nova master: Add test coverage of existing os-volumes-attachments policies https://review.opendev.org/709929
13:36:38	openstackgerrit	Merged openstack/nova master: Introduce scope_types in os-volumes-attachments policy https://review.opendev.org/709388
13:36:51	openstackgerrit	Merged openstack/nova master: Introduce scope_types in os-aggregates policy https://review.opendev.org/701652
13:48:57	openstackgerrit	Lee Yarwood proposed openstack/nova master: WIP/DNM zuul: Attempt to migrate and break up the nova-live-migration job https://review.opendev.org/711604
13:53:54	gmann	brinzhang: getting coffee. I will check the error. is it instance action one ?
14:02:46	brinzhang	gmann: yes, it's the failed patch https://review.opendev.org/#/c/706470/
14:05:00	brinzhang	gmann: the failed is the non-admin expected response is not match the actual response, and the 'traceback' in the actual response, it's should return when the user is non-admin
14:05:22	brinzhang	gmann: thanks for your check.
14:06:27	gmann	ok
14:06:42	brinzhang	johnthetubaguy: thanks +A for that rebase patch
14:10:22	kashyap	bauzas: stephenfin: Heya, for allocating huge pages, have we already considered the existing libvirt API allocPages()? (I don't see it in the source from a lazy `grep`)
14:10:26	kashyap	Documentation: https://libvirt.org/html/libvirt-libvirt-host.html#virNodeAllocPages
14:13:13	openstack	bug 1862633 in OpenStack Compute (nova) "unshelve leak allocation if update port fails" [Medium,Fix released] https://launchpad.net/bugs/1862633 - Assigned to Balazs Gibizer (balazs-gibizer)
14:13:13	openstackgerrit	Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug 1862633 https://review.opendev.org/711626
14:16:41	bauzas	kashyap: good question, I dunno
14:17:56	kashyap	bauzas: Okay, I'll make a mental note to explore it.
14:22:33	stephenfin	kashyap: What's the context?
14:22:57	kashyap	stephenfin: I was triaging a downstream bug, where Nova ends up scheduling a VM to a host w/ insufficient huge pages
14:23:10	kashyap	Therefore it fails to start on the dest host with:
14:23:31	kashyap	[quote]
14:23:32	kashyap	qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/22-instance-00000688,share=yes,size=17179869184,host-nodes=0,policy=bind: unable to map backing store for guest RAM: Cannot allocate memory
14:23:36	kashyap	[/quote]
14:23:57	stephenfin	kashyap: My guess is that they have something else on the host using hugepages and have configured the reserved hugepages config opt
14:24:42	kashyap	stephenfin: So maybe we should get the huge pages usage from the host go from there
14:24:50	kashyap	(And also the Nova config attribute you mention)
14:24:53	kashyap	Thanks for the idea
14:24:59	stephenfin	kashyap: '[DEFAULT] reserved_huge_pages', btw
14:25:12	johnthetubaguy	I have seen that when the NUMA config was bad, by accident
14:25:37	johnthetubaguy	i.e. not enough huge pages on a single numa node, as that is what I implicitly requested
14:26:03	stephenfin	johnthetubaguy: But we do track hugepages on a per-node basis, right?
14:26:40	stephenfin	so there must be something on that node outside of nova consuming those hugepages
14:26:50	stephenfin	or our tracking is broken
14:27:11	johnthetubaguy	I guess, I thought I saw that for that case too, but I kinda stopped digging for other reasons
14:27:21	johnthetubaguy	it was a while back mind
14:27:21	kashyap	So, Nova does something bespoke to account for memory? (/me should read the code)
14:27:58	kashyap	johnthetubaguy: What do you mean "bad NUMA config"? What is a "good one", in Nova's parlance?
14:28:10	stephenfin	kashyap: Look for references to NUMAPagesTopology
14:28:17	stephenfin	we use that object to do our tracking
14:28:33	johnthetubaguy	its more that if you don't specify it, you are requesting a single numa node with all the RAM
14:28:58	johnthetubaguy	if you are trying to request the whole node, which is two numa nodes, that is going to fail
14:29:30	openstackgerrit	Balazs Gibizer proposed openstack/nova stable/stein: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/711629
14:29:39	kashyap	stephenfin: Noted; thanks for the pointer.
14:29:40	stephenfin	so 64GB RAM from a two node host with exactly 64GB total (32GB per node)?
14:30:03	johnthetubaguy	yeah, something like that
14:30:05	stephenfin	gibi++ Thanks, btw :)
14:30:30	stephenfin	brinzhang: looks like johnthetubaguy beat me to it
14:31:37	johnthetubaguy	stephenfin: gmann has added some nice extra policy testing for you to take a peak at btw: https://review.opendev.org/#/c/707039/4/nova/tests/unit/policies/test_create_backup.py
14:32:00	johnthetubaguy	basically test what happen after we lift all the deprecations, at some point in the future (or via deployer config)
14:33:07	kashyap	stephenfin: Also, isn't it tricky (read: "needs bespoke script") to get the all the mapped huge pages usage by different applications on the host?
14:33:18	gmann	brinzhang: johnthetubaguy seems we passed extra bit of project id here - https://review.opendev.org/#/c/710411/4/nova/api/openstack/compute/instance_actions.py@166
14:33:34	gmann	that rule was admin only and does not need project_id
14:34:07	johnthetubaguy	gmann: good catch, should by empty dictionary there

Earlier   Later