| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-19 | |||
| 13:58:28 | dpawlik | kashyap, mriedem: working on it | |
| 14:00:27 | kashyap | Also, this should be called, 'kvm_hypervisor_id_hidden' -- maybe a tad too lengthy, but damn clear than "kvm_hidden", which will make people think: "Oh, this will disable KVM completely!" | |
| 14:00:39 | dansmith | I think it should not say kvm at all | |
| 14:01:07 | mriedem | hide_hypervisor_id | |
| 14:01:09 | mriedem | maybe | |
| 14:02:13 | kashyap | dansmith: It is hiding the KVM signature -- "KVMKVMKVM\0\0\0" | |
| 14:02:33 | dansmith | kashyap: sure but we don't want to end up with a xen_hide_id flag separate | |
| 14:02:39 | kashyap | I also say mentioning it useful because, there's upstream QEMU work to also add "TCG" disabling, too | |
| 14:04:39 | kashyap | dansmith: Yeah, I see what you mean, at Nova-level, you want a generic conf, that can handle different hypervisors (let they be KVM, TCG, Xen, etc) | |
| 14:04:49 | dansmith | yes | |
| 14:05:10 | kashyap | You're correct, mriedem's 'hide_hypervisor_id' sounds best to me | |
| 14:07:47 | sdague | or secret_agent_mode :) | |
| 14:08:40 | kashyap | sdague: Heh | |
| 14:09:02 | kashyap | sdague: You mean, enable_backdoor | |
| 14:10:41 | sfinucan | Morning, mriedem. Could you take a shot at [1] again today? https://review.openstack.org/#/q/status:open+project:openstack/nova+topic:bp/move-nova-cmds-to-cliff | |
| 14:11:44 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 14:11:47 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 14:11:52 | mriedem | maybe | |
| 14:11:52 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 14:12:01 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 14:12:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 14:12:06 | mriedem | sdague: oomichi: mtreinish: i just noticed this bug again https://bugs.launchpad.net/nova/+bug/1634668 | |
| 14:12:06 | openstack | Launchpad bug 1634668 in OpenStack Compute (nova) "Compute API Reference - version response doesn't require 'updated'" [Medium,Confirmed] | |
| 14:12:19 | sfinucan | maybe will do | |
| 14:12:22 | mriedem | about how tempest checks that 'updated' is in the version response, and that test is part of defcore | |
| 14:12:38 | mriedem | would it be wrong to remove that from the tempest schema response validation? | |
| 14:13:58 | kashyap | mdbooth: mriedem: An unrelated question: When trying to migrate 50+ guests, will Nova attempt to do them all in parallel? | |
| 14:15:19 | mdbooth | kashyap: I know my script explicitly pipelined them. Don't recall any limiting in Nova. | |
| 14:15:38 | kashyap | mdbooth: Which script is this? | |
| 14:16:07 | openstackgerrit | Alex Szarka proposed openstack/nova master: Refactor create_delete_server_with_instance_update https://review.openstack.org/466296 | |
| 14:16:37 | mriedem | kashyap: live migration or cold migration, because for live migration https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L526 | |
| 14:17:44 | kashyap | Live migration | |
| 14:17:45 | mriedem | kashyap: by default live migrations are restricted to one at a time on a given compute | |
| 14:17:53 | mriedem | see the max_concurrent_live_migrations option | |
| 14:18:25 | mriedem | sdague: johnthetubaguy: dansmith: oh seers of nova lore, do you remember if boot from volume came before volume attach or after? | |
| 14:18:48 | dansmith | after I'm sure | |
| 14:18:56 | mriedem | that's what i assumed | |
| 14:21:04 | openstackgerrit | Alex Szarka proposed openstack/nova master: Refactor create_delete_server_with_instance_update https://review.openstack.org/466296 | |
| 14:22:35 | kashyap | mriedem: Thanks; that's useful | |
| 14:22:36 | kashyap | "(Integer) Maximum number of live migrations to run concurrently. This limit is enforced to avoid outbound live migrations overwhelming the host/network and causing failures. It is not recommended that you change this unless you are very sure that doing so is safe and stable in your environment." | |
| 14:22:37 | openstackgerrit | Alex Szarka proposed openstack/nova master: Add method for verify multiple notifications https://review.openstack.org/465526 | |
| 14:40:26 | dansmith | woot | |
| 14:40:33 | dansmith | live migration job is passing with multicell too | |
| 14:42:19 | mriedem | that's unpossible | |
| 14:42:34 | dansmith | it's definitely unexpected | |
| 14:45:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate scheduler trusted filter https://review.openstack.org/463065 | |
| 14:48:50 | mriedem | if people are happy with this caching thing we should get it in and start backports https://review.openstack.org/#/c/465042/ | |
| 14:50:21 | openstackgerrit | Daniel Pawlik proposed openstack/nova master: Add hide hypervisor id on guest host https://review.openstack.org/459753 | |
| 14:50:56 | _dpawlik | mriedem, dansmith, kashyap: I made some changes in PS https://review.openstack.org/#/c/459753/ . I must go. If something is bad, I try to fix it later. Thanks for reviewing | |
| 14:51:04 | _dpawlik | have a nice weekend | |
| 15:38:44 | cfriesen | are there any plans to remove the glance proxy in the nova API? | |
| 15:40:48 | artom | cfriesen, it's not already gone? | |
| 15:41:26 | artom | cfriesen, deprecated, I should say, as of https://docs.openstack.org/developer/nova/api_microversion_history.html#id33 | |
| 15:41:40 | artom | cfriesen, but it has to remain in the code for older microversions to work | |
| 15:45:47 | johnthetubaguy | mriedem_afk I would have bet on it being after too, I kinda remember alaski attempting to tidy up a bunch of it some time after it landed | |
| 15:47:45 | alaski | tidy up what now? | |
| 15:51:42 | cfriesen | artom: whoops, missed that. | |
| 15:59:21 | dansmith | alaski: boot from volume I think | |
| 16:01:56 | alaski | ahh. Tidy up is being generous. I just made it actually functional for xenserver, and a little more reliable. | |
| 16:02:57 | alaski | you know what would really help, uuids for bdms :) | |
| 16:04:19 | cfriesen | boot from volume definitely came after volume attach/detach | |
| 16:04:53 | cfriesen | (looking at git history) | |
| 16:39:01 | openstackgerrit | Dan Smith proposed openstack/nova master: De-complicate some of the instance delete path https://review.openstack.org/458636 | |
| 16:39:01 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix live migration devstack hook for multicell environment https://review.openstack.org/458537 | |
| 16:39:02 | openstackgerrit | Dan Smith proposed openstack/nova master: Target cell on local delete https://review.openstack.org/464059 | |
| 18:09:49 | cfriesen | dansmith: you around? | |
| 18:13:47 | dansmith | kinda | |
| 18:21:04 | cfriesen | dansmith: if you remember we were talking about the cost of processing network-changed events. it turns out the neutron work to process those events during a migration doubled in Newton when commit 5de902a went in. That was submitted to fix evacuate bugs, but it meant both source and dest compute nodes got the "network-changed" events. | |
| 18:22:36 | cfriesen | I'm writing up a post, but even after mriedem's optimization at https://review.openstack.org/#/c/465787 I'm seeing 4 simultaneous live migrations push neutron up to using 1300% cpu on the controller, with the spike lasting ~45sec | |
| 18:34:38 | dansmith | 128 neutron api calls peg 13 cores for almost a minute? | |
| 18:37:13 | cfriesen | no, more like a few thousand neutron API calls | |
| 18:37:39 | cfriesen | for each port nova queries the fixed IPs, floating IPs, subnets, etc | |
| 18:37:51 | dansmith | oh I see | |
| 18:40:32 | mriedem | cfriesen: so https://review.openstack.org/#/c/465787 does help? | |
| 18:41:41 | cfriesen | mriedem: yes, hugely. for one migration it cut down on the number of neutron API calls by a factor of 6 or so | |
| 18:41:49 | mriedem | wow nice | |
| 18:42:05 | mriedem | ok so i suppose i need to tidy that up | |
| 18:42:15 | mriedem | i can make it smaller by splitting the refactor part into a separate change | |
| 18:42:47 | cfriesen | mriedem: dansmith: I just posted to the openstack-dev list: "massive overhead processing "network-changed" events during live migration". Some hard data in there. | |
| 18:43:08 | cfriesen | mriedem: I also included the change I proposed to query just the one port rather than listing all of them. | |
| 18:47:40 | mriedem | ok i'm post-alient and post-burrito so i need to get my bearings here | |
| 18:50:18 | cfriesen | mridem: (I'm talking about the optimization I proposed in https://review.openstack.org/#/c/465787....I applied it for my testing) | |
| 18:50:43 | mriedem | oh i haven't seen comments | |
| 18:51:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove glusterfs volume driver https://review.openstack.org/463992 | |
| 18:52:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: remove scality volume driver https://review.openstack.org/449364 | |
| 18:52:10 | cfriesen | mriedem: my one comment about the number of neutron API calls isn't quite accurate, that's actually the number of lines in the neutron API log...but most API calls resulted in one log line so it should be in the ballpark. | |
| 19:00:02 | mriedem | cfriesen: does the list_ports thing really change much? | |
| 19:00:16 | mriedem | replied to your comments | |
| 19:00:21 | mriedem | either way it's a single API call to neutron | |
| 19:00:45 | mriedem | but if we don't have a nw info cache for the instance, for whatever reason - that shouldn't actually happen, we'd need to go back and pull the entire list of ports to rebuild the entire nw info cache | |
| 19:01:20 | cfriesen | mriedem: that list-ports is getting all the ports in the cloud, right? or is it scoped to the project? | |
| 19:01:27 | mriedem | see the search opts | |
| 19:01:35 | mriedem | it's scoped to the ports owned by that tenant associated with that specific instance | |
| 19:01:52 | mriedem | if we just did show_port w/o those filter parameters, it'd be a possible bug | |
| 19:02:04 | mriedem | because the port might no longer be associated with that instance | |
| 19:02:08 | mriedem | which messes up the logic that comes later | |
| 19:02:26 | cfriesen | mriedem: okay, fair enough. givien that it's associated with the instance it's probably fine. I missed that part. | |
| 19:02:31 | mriedem | ok | |
| 19:03:01 | mriedem | _build_network_info_model is really used for 3 different cases, so it gets really hairy logic-wise | |
| 19:03:16 | mriedem | at some point i think we could split it apart, so we didn't have kwarg soup | |