Posted	Nick	Remark
#openstack-nova - 2020-07-17
02:30:28	mnaser	yep, just ran that and its eating up a lot of cpu time
02:30:34	sean-k-mooney	that might be trying to do a revers dns lookup
02:31:24	mnaser	# ip addr | grep inet | wc -l => 746
02:31:40	sean-k-mooney	am im going to have to call it a night but ill be around in a coule of hour.
02:31:46	sean-k-mooney	oh all the link local addresses
02:31:51	sean-k-mooney	on all the tap devices
02:32:10	sean-k-mooney	mnaser: artom saw an issue with this before
02:32:17	mnaser	len(netifaces.interfaces()) => 973
02:32:26	mnaser	yeah, i think i'm onto something here
02:32:28	sean-k-mooney	i think he fixed it but maybe not on master
02:32:33	sean-k-mooney	* stien
02:32:44	sean-k-mooney	maybe we didnt backport it
02:34:40	mnaser	t = time.time(); netifaces.ifaddresses('qvo64d61663-06'); print(time.time() - t) => 133ms
02:35:02	mnaser	973*133ms = 129s
02:35:48	sean-k-mooney	https://review.opendev.org/#/c/672161/
02:35:51	mnaser	we should probably just skip qbr/qvo/qvb/tap in there
02:36:20	sean-k-mooney	ya proably
02:36:46	sean-k-mooney	so artom moved it to init_host to fix https://bugs.launchpad.net/nova/+bug/1837075
02:36:46	openstack	Launchpad bug 1837075 in OpenStack Compute (nova) stein "Evacuation takes too long when destination host has a large number of NICs" [Low,Fix committed] - Assigned to Artom Lifshitz (notartom)
02:36:59	mnaser	but instead it takes a long time to init_host :)
02:37:28	sean-k-mooney	yes but we used to call it at runtime several times
02:37:37	sean-k-mooney	so we look it up once now and cache it
02:38:24	sean-k-mooney	we used to call it every time we called get_host_ip_addr
02:38:31	sean-k-mooney	which we did for all move operations
02:38:31	mnaser	yeah, perhaps that function could use skipping nova-managed devices
02:39:00	sean-k-mooney	this is all its used for
02:39:02	sean-k-mooney	ips = compute_utils.get_machine_ips()
02:39:04	sean-k-mooney	if CONF.my_ip not in ips:
02:39:06	sean-k-mooney	LOG.warning('my_ip address (%(my_ip)s) was not found on '
02:39:08	sean-k-mooney	'any of the interfaces: %(ifaces)s',
02:39:10	sean-k-mooney	{'my_ip': CONF.my_ip, 'ifaces': ", ".join(ips)})
02:39:19	sean-k-mooney	it validating that the ip we set in the config if its set is on the host
02:39:29	sean-k-mooney	there is proably a better way to do that
02:40:36	mnaser	yeah i cant find a better thing but i think skipping qbr/qvo/qvb/tap might speed it up.. a lot
02:40:56	sean-k-mooney	where you debuging startup latency
02:41:11	sean-k-mooney	or were you debuging an ongoing issue with new vms
02:41:22	sean-k-mooney	this will be a nice fix but it wont affect the later
02:41:53	mnaser	yeah i agree it wont affect the latter, i just switched to native and running the 'benchmark' now against the cloud
02:43:32	sean-k-mooney	ok well if it helps or not let me know. ill be back in 8 hours or so after i get some sleep
02:43:34	sean-k-mooney	o/
02:44:10	mnaser	yeah, i think i'll leave it to get soak tested overnight with those changes
02:44:14	mnaser	but we're onto something ,thanks sean-k-mooney
02:45:14	sean-k-mooney	no worries
04:21:07	openstackgerrit	Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485
04:21:08	openstackgerrit	Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933
04:21:08	openstackgerrit	Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
04:21:09	openstackgerrit	Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997
04:21:09	openstackgerrit	Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395
06:07:24	openstackgerrit	Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485
06:07:25	openstackgerrit	Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933
06:07:25	openstackgerrit	Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
06:07:26	openstackgerrit	Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997
06:07:26	openstackgerrit	Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395
06:12:15	openstackgerrit	Wenping Song proposed openstack/nova-specs master: Add no user token when get Cyborg client https://review.opendev.org/740184
07:38:11	openstackgerrit	Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439
07:45:12	openstackgerrit	Luigi Toscano proposed openstack/nova stable/ussuri: zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full https://review.opendev.org/741590
08:14:14	openstackgerrit	Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439
08:37:05	openstackgerrit	Balazs Gibizer proposed openstack/nova-specs master: Support SRIOV interface attach and detach https://review.opendev.org/741601
09:02:03	openstackgerrit	Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration https://review.opendev.org/741529
09:03:17	openstackgerrit	Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Send announce-self post live migration https://review.opendev.org/741529
09:04:44	tobias-urdin	sean-k-mooney: yes I agree, it's a dirty workaround but until it is solved I will need to carry this downstream
09:05:15	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Remove broken legacy zuul jobs https://review.opendev.org/741603
09:07:02	gibi	gmann, tosky: I proposed some clean of our jobs ^^
09:08:20	tosky	gibi: as long as it works for you, I'm fine :)
09:08:28	gibi	tosky: they never passed
09:08:41	gibi	tosky: and they are really old jobs
09:09:07	gibi	and anyhow if somebody needs them and cares about them then we can resurrect them in zuul v3 form later
09:09:30	tosky	I don't remember if I mentioned it yesterday: even though it's not part of the main goal (just stretch), if you plan to backport the fixes to the older branches, that will help cleaning up openstack-zuul-jobs
09:09:59	tosky	that's why I proposed a backport of the fix that was merged yesterday
09:09:59	gibi	tosky: I'm happy to backport these job deletes to older branches
09:10:05	tosky	thanks :)
09:23:30	gibi	lyarwood, melwitt: if you have time could you hit this pike backport? https://review.opendev.org/#/q/topic:bug/1862633+status:open
09:52:18	lyarwood	gibi: ack I can look today
09:52:35	gibi	thanks
09:55:11	gibi	sean-k-mooney, stephenfin: fyi I pushed a small spec for SRIOV attach https://review.opendev.org/#/c/741601/
10:22:21	openstackgerrit	Wenping Song proposed openstack/nova master: Cyborg shelve/unshelve support https://review.opendev.org/729563
10:26:52	brinzhang0	gibi: hi, I update the nova runway, added the cyborg-shelve-and-unshelve patches to the queue
10:46:51	stephenfin	gibi: I lost the race for some o.vo versions. Could I get your +2 on this again? https://review.opendev.org/#/c/739210/
10:51:29	stephenfin	gibi: -1 on spec for nits but otherwise perfect
10:53:05	openstackgerrit	Stephen Finucane proposed openstack/nova master: libvirt: Re-enable live snapshot for paused instances https://review.opendev.org/740334
10:53:06	openstackgerrit	Stephen Finucane proposed openstack/nova master: libvirt: Remove workaround for really old QEMU https://review.opendev.org/740335
10:53:06	openstackgerrit	Stephen Finucane proposed openstack/nova master: libvirt: Use better variable names for '_create_guest' https://review.opendev.org/740945
10:53:07	openstackgerrit	Stephen Finucane proposed openstack/nova master: WIP: Attempt to restore some sanity to snapshot https://review.opendev.org/740336
10:53:51	stephenfin	lyarwood: The first two of those are pretty simple but would benefit from your eyes, IMO. The third is a FUP for your comments ^
10:56:38	kashyap	stephenfin: Good catch on re-enabling the 'paused' thing
10:56:40	kashyap	I recall the old change
10:57:25	kashyap	Yep, it makes sense, given how far we've come w/ newer (and largely improved) versions
11:37:06	sean-k-mooney	gibi: ill read your sepc shortly. did you see my comments on the code change.
11:37:47	sean-k-mooney	your code change wont fix PF detach either so i might just add a small patch on top to do that
12:04:10	openstackgerrit	Stephen Finucane proposed openstack/nova master: api: Add type hints for 'nova.compute.api' https://review.opendev.org/741623
12:04:11	openstackgerrit	Stephen Finucane proposed openstack/nova master: WIP: api: Add 'nova.compute.API.migrate' https://review.opendev.org/741624
12:05:47	openstackgerrit	Elod Illes proposed openstack/nova stable/queens: Reject boot request for unsupported images https://review.opendev.org/708609
12:21:03	openstackgerrit	Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561
13:01:09	jkulik	How often and how do you clean up your instances table in the cell0 db?
13:01:52	sean-k-mooney	i think you do it with nova-manage
13:02:13	sean-k-mooney	how often is up to you
13:02:29	sean-k-mooney	i think downstream we default to every 30days
13:03:52	jkulik	hm ... I thought so, but I can't seem to find the docs or the code for that
13:05:10	sean-k-mooney	i think you do "nova-manage db archive_deleted_rows" then "nova-manage db purge"
13:05:28	sean-k-mooney	but you need to us a config file with the cell0 credentials

Earlier   Later