Posted	Nick	Remark
#openstack-nova - 2019-01-24
15:25:59	BjoernT	Hello is there anyone here familiar with instance store baked by NFS and larger volume, as I see performance issues with the image cache manager ?
15:26:08	BjoernT	https://bugs.launchpad.net/nova/+bug/1804262/ is the original bug I had opened
15:26:09	openstack	Launchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Undecided,New]
15:26:20	openstackgerrit	Matt Riedemann proposed openstack/nova master: Skip checking of target_dev for vhostuser https://review.openstack.org/627540
15:29:43	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421
15:29:43	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
15:29:44	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Add pf_interface_name tag to passthrough_whitelist https://review.openstack.org/625311
15:29:44	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Record requester in the InstancePCIRequest https://review.openstack.org/625310
15:29:45	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Ensure that bandwidth and VF are from the same PF https://review.openstack.org/623543
15:31:13	stephenfin	alex_xu: Regarding https://review.openstack.org/#/c/631038/3/nova/virt/libvirt/driver.py@6328
15:31:31	stephenfin	alex_xu: Is it not a reasonable assumption that there will be some small pages on each NUMA node?
15:31:45	stephenfin	Far as I know, you can't boot a system with only large/huge pages
15:33:02	sean-k-mooney	stephenfin: correct you cant
15:34:01	sean-k-mooney	also event if there are 0 available i think you will still have the mempage listed
15:36:28	stephenfin	sean-k-mooney++
15:49:17	jangutter	stephenfin: allocating everything to hugepages is the quickest way to get a system unbootable by hitting the OOM killer.
15:49:57	sean-k-mooney	jangutter: yep the kernel cant allocate its own memory form hugepages so it gets grumpy
15:51:14	stephenfin	sean-k-mooney: When we move hugepage tracking to placement, we should get rid of this thing https://review.openstack.org/#/c/631053/1/nova/virt/hardware.py@1059
15:51:45	stephenfin	(the second part of your comment, that is
15:51:46	stephenfin	)
15:52:06	sean-k-mooney	stephenfin we can only do that if we track all memory as meory of a set size
15:52:40	sean-k-mooney	so we would need to get rid of memory_mb from the compute node and always afine guest to numa nodes
15:54:14	sean-k-mooney	i personally woudl prefer to do that but i dont think we can just remove it simply
15:54:22	sean-k-mooney	maybe we can
15:54:25	stephenfin	Hmm that's a good point
15:55:00	sean-k-mooney	honestly i would prefer to have a singel code path used for dedicate/shared cpus and all mempages sizes
15:55:15	sean-k-mooney	instead of the 2 we have now for numa guests and non numa guests
16:07:28	bauzas	folks, FWIW, I'm on a conference today and tomorow named 'SnowCamp' where I'm an organizer
16:07:31	bauzas	mriedem: ^
16:12:07	mriedem	mdbooth: fyi image cache manager perf bug https://bugs.launchpad.net/nova/+bug/1804262/ - i put some thoughts on how to optimize where they are hitting timeouts
16:12:08	openstack	Launchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Medium,Triaged]
16:13:17	BjoernT	Thanks for the feedback, I did not know about the GMR report (good old amiga times reflect back on me, lol)
16:14:25	mriedem	BjoernT: i think i've sorted out the issue though so the gmr isn't really necessary
16:14:45	BjoernT	ok I'm still curious what it looks like
16:14:48	mriedem	you're just timing out trying to pull bdms for 705 instances from the conductor within 60 seconds (default rpc_response_timeout)
16:15:02	BjoernT	took me a day to actually make that association with failing service checkins and the image cache manager
16:15:20	BjoernT	we run rpc_response timeout upwards of 300 already
16:15:30	BjoernT	now even larger
16:15:35	mriedem	rpc_response_timeout is used mostly globally so tweaking it for one thing does not surprise me that it causes side effects elsehwere
16:15:50	BjoernT	yeah I know
16:16:01	mriedem	we also have a long_rpc_timeout since rocky, but you're on an older release than that, and this code / query isn't using that timeout
16:16:27	BjoernT	I had one workaround in mind to shutoff image cache manger globally except one service node, which would not be used for scheduling
16:16:43	BjoernT	ok, I'm still stuck on pike but good to know
16:17:07	mriedem	i could maybe hack something up here and you can take a look at it
16:17:38	BjoernT	yes I should be able to test that in that environment once I get signoff
16:22:01	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Reject interface attach with QoS aware port https://review.openstack.org/570078
16:22:02	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Create RequestGroup from neutron port https://review.openstack.org/625941
16:22:02	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079
16:22:03	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Read port resource request from Neutron https://review.openstack.org/630720
16:22:03	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Include requested_resources to allocation candidate query https://review.openstack.org/625942
16:22:04	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Reject resize with port having resource request https://review.openstack.org/630722
16:22:04	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Reject server create with port having resource request https://review.openstack.org/630721
16:22:05	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Reject evacuate with port having resource request https://review.openstack.org/630724
16:22:05	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Reject migrate with port having resource request https://review.openstack.org/630723
16:22:06	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268
16:22:06	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Reject unshelve with port having resource request https://review.openstack.org/630725
16:22:07	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239
16:22:07	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527
16:22:08	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240
16:22:08	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528
16:22:09	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459
16:22:09	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529
16:22:10	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
16:22:15	gibi	I'm sorry
16:22:22	sean-k-mooney	:)
16:26:12	gibi	mriedem: I've fixed your comments in https://review.openstack.org/#/c/570078
16:27:17	melwitt	mriedem, efried_2dmtg: would one of y'all be able to run the nova meeting next week? I will be traveling for work and unable to run it
16:29:24	mriedem	yes i can
16:31:56	mriedem	gibi: lgtm, +2
16:33:42	gibi	mriedem: thanks
16:33:52	gibi	the next in that series is pretty similar
16:34:27	melwitt	mriedem: great, thanks
16:34:32	mriedem	ok in the tab queue
16:34:49	gibi	coll
16:34:52	gibi	cool
17:03:47	cfriesen	artom: stephenfin: any news on the live migration stuff? I'm happy to help test experimental code.
17:04:37	artom	cfriesen, heya, afraid not, was mostly downstream this week. I need to get back to it and put up stuff, even WIP
17:15:44	cfriesen	I totally get the downstream stuff. :) Story of my life.
17:22:36	mriedem	BjoernT_: i should have a patch up shortly, just running final tests
17:32:19	openstackgerrit	Matt Riedemann proposed openstack/nova master: Chunk BDM queries from ImageCacheManager._list_running_instances() https://review.openstack.org/633042
17:32:20	mriedem	BjoernT_: ^
17:37:18	openstackgerrit	Matt Riedemann proposed openstack/nova master: Chunk BDM queries from ImageCacheManager._list_running_instances() https://review.openstack.org/633042
17:37:28	openstackgerrit	melanie witt proposed openstack/nova master: Raise 403 instead of 500 error from attach volume API https://review.openstack.org/632904
17:37:29	openstackgerrit	melanie witt proposed openstack/nova master: Add configuration of maximum disk devices to attach https://review.openstack.org/616777
17:40:48	mriedem	kashyap: seeing a weird issue in the gate with libvirt 4.0.0 where sometimes an instance delete times out in libvirt and it looks like there is a loop in libvirt that is busted https://bugs.launchpad.net/nova/+bug/1813192
17:40:49	openstack	Launchpad bug 1813192 in OpenStack Compute (nova) "libvirt: instance delete fails with "Cannot destroy instance, operation time out: libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)" in bionic nodes (libvirt 4.0.0, qemu 2.11)" [Medium,Confirmed]
17:40:58	kashyap	mriedem: Haha, eerie you ping me!
17:41:10	kashyap	mriedem: I was _just_ writing a nice message for you to ask for help w/ some unit test debugging
17:41:14	mriedem	only seeing this on ubuntu bionic
17:41:32	kashyap	Oh, the "cannot acquire state change lock" error.
17:41:50	kashyap	Hmm, it is unofficially the most difficult one to debug.
17:42:14	mriedem	only 3 hits in 7 days so not a huge issue
17:42:26	kashyap	Will check the logs once I have dinner. I was in back-to-back meeting hell
17:42:42	kashyap	Thanks for the heads-up
17:43:28	kashyap	I was about to ping you to borrow some eyes on these unit test failures on the version bump WIP patch: https://review.openstack.org/#/c/632507/
17:43:47	kashyap	But the patch is not fully done; so I don't want to waste your time clicking on it. But I at least reduced the failures from 136 to 5, though.
17:44:10	kashyap	And I spent a crap load of time w/ PDB yesterday & some time today. Was still duking around.
17:45:31	kashyap	On the off chance you have a minute, any comments / snide remarks on the review are welcome.
17:45:47	mriedem	if tempest is failing you've got bigger problems than just unit tests
17:46:42	mriedem	something is busted with console logs http://logs.openstack.org/07/632507/3/check/tempest-full/73a00be/controller/logs/screen-n-cpu.txt.gz?level=TRACE

Earlier   Later