| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-17 | |||
| 02:23:34 | mnaser | the plugs that happen on start up were also not very fast, taking ~5s each | |
| 02:23:44 | mnaser | perhaps this will show difference | |
| 02:24:11 | mnaser | 2020-07-17 02:23:44.773 1550960 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: entering ACTIVE _transition /openstack/venvs/nova-19.0.8/lib/python2.7/site-packages/ovs/reconnect.py:485 | |
| 02:24:13 | sean-k-mooney | ah this is is where that gets generateed https://github.com/openstack/os-vif/blob/master/os_vif/plugin.py#L79 | |
| 02:24:51 | sean-k-mooney | mnaser: that looks like its connect with the native backend alright | |
| 02:25:10 | mnaser | yeah but nova-compute is pegged at 100% cpu now on start up | |
| 02:25:14 | mnaser | time to see what its doing | |
| 02:25:29 | sean-k-mooney | well it does a lot | |
| 02:25:39 | sean-k-mooney | including pluging all the port of every vm on the host | |
| 02:25:59 | sean-k-mooney | but its going to run all the reousce tracker stuff before that form init_host | |
| 02:26:20 | mnaser | just sent USR2 | |
| 02:26:46 | sean-k-mooney | it that the guru meditation report | |
| 02:27:20 | mnaser | yeah | |
| 02:27:36 | sean-k-mooney | ya i have no idea how to read those | |
| 02:27:54 | mnaser | oddly enough though, its still taking 5 seconds to plug a port on start up | |
| 02:28:27 | sean-k-mooney | has the privsep load dropped | |
| 02:29:22 | sean-k-mooney | i dont know if ovsdbapp uses privsep internaly but os-vif nolonger need to use prvisep for the ovs db updates at least | |
| 02:29:27 | mnaser | oooou i have an idea | |
| 02:29:37 | mnaser | i think ipv6 being enabled is hurting this host | |
| 02:29:46 | sean-k-mooney | it still needs privsep for other thngs | |
| 02:29:53 | sean-k-mooney | oh hum | |
| 02:29:53 | mnaser | seems like it was stuck on /openstack/venvs/nova-19.0.8/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:649 in _check_my_ip | |
| 02:29:57 | mnaser | on start up | |
| 02:30:02 | mnaser | im going to time that | |
| 02:30:11 | sean-k-mooney | ya maybe | |
| 02:30:16 | mnaser | specifically `ips = compute_utils.get_machine_ips()` | |
| 02:30:28 | mnaser | yep, just ran that and its eating up a lot of cpu time | |
| 02:30:34 | sean-k-mooney | that might be trying to do a revers dns lookup | |
| 02:31:24 | mnaser | # ip addr | grep inet | wc -l => 746 | |
| 02:31:40 | sean-k-mooney | am im going to have to call it a night but ill be around in a coule of hour. | |
| 02:31:46 | sean-k-mooney | oh all the link local addresses | |
| 02:31:51 | sean-k-mooney | on all the tap devices | |
| 02:32:10 | sean-k-mooney | mnaser: artom saw an issue with this before | |
| 02:32:17 | mnaser | len(netifaces.interfaces()) => 973 | |
| 02:32:26 | mnaser | yeah, i think i'm onto something here | |
| 02:32:28 | sean-k-mooney | i think he fixed it but maybe not on master | |
| 02:32:33 | sean-k-mooney | * stien | |
| 02:32:44 | sean-k-mooney | maybe we didnt backport it | |
| 02:34:40 | mnaser | t = time.time(); netifaces.ifaddresses('qvo64d61663-06'); print(time.time() - t) => 133ms | |
| 02:35:02 | mnaser | 973*133ms = 129s | |
| 02:35:48 | sean-k-mooney | https://review.opendev.org/#/c/672161/ | |
| 02:35:51 | mnaser | we should probably just skip qbr/qvo/qvb/tap in there | |
| 02:36:20 | sean-k-mooney | ya proably | |
| 02:36:46 | sean-k-mooney | so artom moved it to init_host to fix https://bugs.launchpad.net/nova/+bug/1837075 | |
| 02:36:46 | openstack | Launchpad bug 1837075 in OpenStack Compute (nova) stein "Evacuation takes too long when destination host has a large number of NICs" [Low,Fix committed] - Assigned to Artom Lifshitz (notartom) | |
| 02:36:59 | mnaser | but instead it takes a long time to init_host :) | |
| 02:37:28 | sean-k-mooney | yes but we used to call it at runtime several times | |
| 02:37:37 | sean-k-mooney | so we look it up once now and cache it | |
| 02:38:24 | sean-k-mooney | we used to call it every time we called get_host_ip_addr | |
| 02:38:31 | sean-k-mooney | which we did for all move operations | |
| 02:38:31 | mnaser | yeah, perhaps that function could use skipping nova-managed devices | |
| 02:39:00 | sean-k-mooney | this is all its used for | |
| 02:39:02 | sean-k-mooney | ips = compute_utils.get_machine_ips() | |
| 02:39:04 | sean-k-mooney | if CONF.my_ip not in ips: | |
| 02:39:06 | sean-k-mooney | LOG.warning('my_ip address (%(my_ip)s) was not found on ' | |
| 02:39:08 | sean-k-mooney | 'any of the interfaces: %(ifaces)s', | |
| 02:39:10 | sean-k-mooney | {'my_ip': CONF.my_ip, 'ifaces': ", ".join(ips)}) | |
| 02:39:19 | sean-k-mooney | it validating that the ip we set in the config if its set is on the host | |
| 02:39:29 | sean-k-mooney | there is proably a better way to do that | |
| 02:40:36 | mnaser | yeah i cant find a better thing but i think skipping qbr/qvo/qvb/tap might speed it up.. a lot | |
| 02:40:56 | sean-k-mooney | where you debuging startup latency | |
| 02:41:11 | sean-k-mooney | or were you debuging an ongoing issue with new vms | |
| 02:41:22 | sean-k-mooney | this will be a nice fix but it wont affect the later | |
| 02:41:53 | mnaser | yeah i agree it wont affect the latter, i just switched to native and running the 'benchmark' now against the cloud | |
| 02:43:32 | sean-k-mooney | ok well if it helps or not let me know. ill be back in 8 hours or so after i get some sleep | |
| 02:43:34 | sean-k-mooney | o/ | |
| 02:44:10 | mnaser | yeah, i think i'll leave it to get soak tested overnight with those changes | |
| 02:44:14 | mnaser | but we're onto something ,thanks sean-k-mooney | |
| 02:45:14 | sean-k-mooney | no worries | |
| 04:21:07 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485 | |
| 04:21:08 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933 | |
| 04:21:08 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | |
| 04:21:09 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997 | |
| 04:21:09 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395 | |
| 06:07:24 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485 | |
| 06:07:25 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933 | |
| 06:07:25 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | |
| 06:07:26 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997 | |
| 06:07:26 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395 | |
| 06:12:15 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Add no user token when get Cyborg client https://review.opendev.org/740184 | |
| 07:38:11 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439 | |
| 07:45:12 | openstackgerrit | Luigi Toscano proposed openstack/nova stable/ussuri: zuul: remove legacy-tempest-dsvm-neutron-dvr-multinode-full https://review.opendev.org/741590 | |
| 08:14:14 | openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439 | |
| 08:37:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Support SRIOV interface attach and detach https://review.opendev.org/741601 | |
| 09:02:03 | openstackgerrit | Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration https://review.opendev.org/741529 | |
| 09:03:17 | openstackgerrit | Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Send announce-self post live migration https://review.opendev.org/741529 | |
| 09:04:44 | tobias-urdin | sean-k-mooney: yes I agree, it's a dirty workaround but until it is solved I will need to carry this downstream | |
| 09:05:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove broken legacy zuul jobs https://review.opendev.org/741603 | |
| 09:07:02 | gibi | gmann, tosky: I proposed some clean of our jobs ^^ | |
| 09:08:20 | tosky | gibi: as long as it works for you, I'm fine :) | |
| 09:08:28 | gibi | tosky: they never passed | |
| 09:08:41 | gibi | tosky: and they are really old jobs | |
| 09:09:07 | gibi | and anyhow if somebody needs them and cares about them then we can resurrect them in zuul v3 form later | |
| 09:09:30 | tosky | I don't remember if I mentioned it yesterday: even though it's not part of the main goal (just stretch), if you plan to backport the fixes to the older branches, that will help cleaning up openstack-zuul-jobs | |
| 09:09:59 | tosky | that's why I proposed a backport of the fix that was merged yesterday | |
| 09:09:59 | gibi | tosky: I'm happy to backport these job deletes to older branches | |
| 09:10:05 | tosky | thanks :) | |
| 09:23:30 | gibi | lyarwood, melwitt: if you have time could you hit this pike backport? https://review.opendev.org/#/q/topic:bug/1862633+status:open | |
| 09:52:18 | lyarwood | gibi: ack I can look today | |
| 09:52:35 | gibi | thanks | |