Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-17
02:24:51 sean-k-mooney mnaser: that looks like its connect with the native backend alright
02:25:10 mnaser yeah but nova-compute is pegged at 100% cpu now on start up
02:25:14 mnaser time to see what its doing
02:25:29 sean-k-mooney well it does a lot
02:25:39 sean-k-mooney including pluging all the port of every vm on the host
02:25:59 sean-k-mooney but its going to run all the reousce tracker stuff before that form init_host
02:26:20 mnaser just sent USR2
02:26:46 sean-k-mooney it that the guru meditation report
02:27:20 mnaser yeah
02:27:36 sean-k-mooney ya i have no idea how to read those
02:27:54 mnaser oddly enough though, its still taking 5 seconds to plug a port on start up
02:28:27 sean-k-mooney has the privsep load dropped
02:29:22 sean-k-mooney i dont know if ovsdbapp uses privsep internaly but os-vif nolonger need to use prvisep for the ovs db updates at least
02:29:27 mnaser oooou i have an idea
02:29:37 mnaser i think ipv6 being enabled is hurting this host
02:29:46 sean-k-mooney it still needs privsep for other thngs
02:29:53 sean-k-mooney oh hum
02:29:53 mnaser seems like it was stuck on /openstack/venvs/nova-19.0.8/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:649 in _check_my_ip
02:29:57 mnaser on start up
02:30:02 mnaser im going to time that
02:30:11 sean-k-mooney ya maybe
02:30:16 mnaser specifically `ips = compute_utils.get_machine_ips()`
02:30:28 mnaser yep, just ran that and its eating up a lot of cpu time
02:30:34 sean-k-mooney that might be trying to do a revers dns lookup
02:31:24 mnaser # ip addr | grep inet | wc -l => 746
02:31:40 sean-k-mooney am im going to have to call it a night but ill be around in a coule of hour.
02:31:46 sean-k-mooney oh all the link local addresses
02:31:51 sean-k-mooney on all the tap devices
02:32:10 sean-k-mooney mnaser: artom saw an issue with this before
02:32:17 mnaser len(netifaces.interfaces()) => 973
02:32:26 mnaser yeah, i think i'm onto something here
02:32:28 sean-k-mooney i think he fixed it but maybe not on master
02:32:33 sean-k-mooney * stien
02:32:44 sean-k-mooney maybe we didnt backport it
02:34:40 mnaser t = time.time(); netifaces.ifaddresses('qvo64d61663-06'); print(time.time() - t) => 133ms
02:35:02 mnaser 973*133ms = 129s
02:35:48 sean-k-mooney https://review.opendev.org/#/c/672161/
02:35:51 mnaser we should probably just skip qbr/qvo/qvb/tap in there
02:36:20 sean-k-mooney ya proably
02:36:46 sean-k-mooney so artom moved it to init_host to fix https://bugs.launchpad.net/nova/+bug/1837075
02:36:46 openstack Launchpad bug 1837075 in OpenStack Compute (nova) stein "Evacuation takes too long when destination host has a large number of NICs" [Low,Fix committed] - Assigned to Artom Lifshitz (notartom)
02:36:59 mnaser but instead it takes a long time to init_host :)
02:37:28 sean-k-mooney yes but we used to call it at runtime several times
02:37:37 sean-k-mooney so we look it up once now and cache it
02:38:24 sean-k-mooney we used to call it every time we called get_host_ip_addr
02:38:31 sean-k-mooney which we did for all move operations
02:38:31 mnaser yeah, perhaps that function could use skipping nova-managed devices
02:39:00 sean-k-mooney this is all its used for
02:39:02 sean-k-mooney ips = compute_utils.get_machine_ips()
02:39:04 sean-k-mooney if CONF.my_ip not in ips:
02:39:06 sean-k-mooney LOG.warning('my_ip address (%(my_ip)s) was not found on '
02:39:08 sean-k-mooney 'any of the interfaces: %(ifaces)s',
02:39:10 sean-k-mooney {'my_ip': CONF.my_ip, 'ifaces': ", ".join(ips)})
02:39:19 sean-k-mooney it validating that the ip we set in the config if its set is on the host
02:39:29 sean-k-mooney there is proably a better way to do that
02:40:36 mnaser yeah i cant find a better thing but i think skipping qbr/qvo/qvb/tap might speed it up.. a lot
02:40:56 sean-k-mooney where you debuging startup latency
02:41:11 sean-k-mooney or were you debuging an ongoing issue with new vms
02:41:22 sean-k-mooney this will be a nice fix but it wont affect the later
02:41:53 mnaser yeah i agree it wont affect the latter, i just switched to native and running the 'benchmark' now against the cloud
02:43:32 sean-k-mooney ok well if it helps or not let me know. ill be back in 8 hours or so after i get some sleep
02:43:34 sean-k-mooney o/
02:44:10 mnaser yeah, i think i'll leave it to get soak tested overnight with those changes
02:44:14 mnaser but we're onto something ,thanks sean-k-mooney
02:45:14 sean-k-mooney no worries
04:21:07 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485
04:21:08 openstackgerrit Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933
04:21:08 openstackgerrit Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
04:21:09 openstackgerrit Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997
04:21:09 openstackgerrit Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395
06:07:24 openstackgerrit Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485
06:07:25 openstackgerrit Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933
06:07:25 openstackgerrit Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
06:07:26 openstackgerrit Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997
06:07:26 openstackgerrit Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395
06:12:15 openstackgerrit Wenping Song proposed openstack/nova-specs master: Add no user token when get Cyborg client https://review.opendev.org/740184
07:38:11 openstackgerrit Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439
07:45:12 openstackgerrit Luigi Toscano proposed openstack/nova stable/ussuri: zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full https://review.opendev.org/741590
08:14:14 openstackgerrit Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439
08:37:05 openstackgerrit Balazs Gibizer proposed openstack/nova-specs master: Support SRIOV interface attach and detach https://review.opendev.org/741601
09:02:03 openstackgerrit Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration https://review.opendev.org/741529
09:03:17 openstackgerrit Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Send announce-self post live migration https://review.opendev.org/741529
09:04:44 tobias-urdin sean-k-mooney: yes I agree, it's a dirty workaround but until it is solved I will need to carry this downstream
09:05:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove broken legacy zuul jobs https://review.opendev.org/741603
09:07:02 gibi gmann, tosky: I proposed some clean of our jobs ^^
09:08:20 tosky gibi: as long as it works for you, I'm fine :)
09:08:28 gibi tosky: they never passed
09:08:41 gibi tosky: and they are really old jobs
09:09:07 gibi and anyhow if somebody needs them and cares about them then we can resurrect them in zuul v3 form later
09:09:30 tosky I don't remember if I mentioned it yesterday: even though it's not part of the main goal (just stretch), if you plan to backport the fixes to the older branches, that will help cleaning up openstack-zuul-jobs
09:09:59 tosky that's why I proposed a backport of the fix that was merged yesterday
09:09:59 gibi tosky: I'm happy to backport these job deletes to older branches
09:10:05 tosky thanks :)
09:23:30 gibi lyarwood, melwitt: if you have time could you hit this pike backport? https://review.opendev.org/#/q/topic:bug/1862633+status:open
09:52:18 lyarwood gibi: ack I can look today
09:52:35 gibi thanks
09:55:11 gibi sean-k-mooney, stephenfin: fyi I pushed a small spec for SRIOV attach https://review.opendev.org/#/c/741601/
10:22:21 openstackgerrit Wenping Song proposed openstack/nova master: Cyborg shelve/unshelve support https://review.opendev.org/729563
10:26:52 brinzhang0 gibi: hi, I update the nova runway, added the cyborg-shelve-and-unshelve patches to the queue
10:46:51 stephenfin gibi: I lost the race for some o.vo versions. Could I get your +2 on this again? https://review.opendev.org/#/c/739210/

Earlier   Later