| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-13 | |||
| 14:35:33 | dansmith | sahid: here's the patch that added it to compute manager: https://review.openstack.org/#/c/558001/10/nova/compute/manager.py | |
| 14:35:53 | dansmith | sahid: which starts the wait *before* the pre-live-on-dest call runs which actually does the plug, so it's not racy | |
| 14:36:33 | dansmith | and you can see, that patch will consume the notification so that your wait in libvirt would never see it | |
| 14:36:52 | sean-k-mooney | dansmith: sahid well reducing the live migration bandwith was really the equvalent of a random sleep(x) as it was never guarneteed to be enough as even with the minium migration bandwith i could still complete before the netwroking was set up | |
| 14:37:34 | dansmith | sean-k-mooney: yeah I said that several times on the initial review :) | |
| 14:39:22 | dansmith | the assertion was made that we could only do it where he had it because of when/where the plug happens, | |
| 14:39:32 | dansmith | and that slowing the migration was the only way to catch it first, | |
| 14:39:38 | dansmith | but that clearly wasn't correct | |
| 14:39:57 | dansmith | so matt's version in compute manager does the full wait before we even start, which is definitely better | |
| 14:40:44 | sahid | how that is supposed to work? linux bridge agent is configuring the network only when it see the new tap device created | |
| 14:41:17 | sahid | that is basically why i took that way to fix the issue, libvirt is creating the tap device when calling migrateToURI | |
| 14:41:18 | dansmith | right, but that happens when we plug the vif, yeah? | |
| 14:41:27 | sahid | no | |
| 14:41:42 | dansmith | sahid: did you look at the bug reported by the guy on pike? | |
| 14:41:57 | dansmith | the event came *before* we even started listening for it | |
| 14:42:00 | sean-k-mooney | dansmith: for linux bridge no. we we do plug we call brctl add-interface bridge but the tap does not exist at that point | |
| 14:42:39 | dansmith | sean-k-mooney: do you mean add-br? | |
| 14:43:09 | sean-k-mooney | no ill get the link to the code in os-vif | |
| 14:43:16 | gibi | tetsuro: depending of the size of the confict I will do the rebase tomorrow | |
| 14:44:02 | dansmith | sean-k-mooney: sahid you should also look at matt's comments on the CI test patch where he follows the sequence of events in a CI run which was configured in such a way that we'd hit the timeout | |
| 14:44:06 | sean-k-mooney | actully wait for linux bridge plugin plug jsut creates teh bridge if it does not exist | |
| 14:44:32 | dansmith | sean-k-mooney: did you look at the bug where we can see the event coming *before* we start the live migration? | |
| 14:44:53 | openstack | Launchpad bug 1786346 in OpenStack Compute (nova) rocky "live migrations slow" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 14:44:53 | dansmith | https://bugs.launchpad.net/nova/+bug/1786346 | |
| 14:45:07 | dansmith | and since that happens and we don't stop the migration, | |
| 14:45:18 | dansmith | the migration continues for a really long time at 1MB/s | |
| 14:45:27 | tetsuro | gibi: Sure... and feel free to ask me to do that. I'll check the IRC log, anyway. | |
| 14:45:28 | dansmith | the event did arrive, but before we were listening for it | |
| 14:45:34 | dansmith | so we failed unnecessarily, | |
| 14:45:41 | dansmith | and then the migration ran "forever" | |
| 14:46:09 | sean-k-mooney | actully if we pass the interface it does add it https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L149 its been a while since i looked at this code | |
| 14:46:11 | gibi | tetsuro: OK | |
| 14:46:26 | dansmith | sean-k-mooney: during plug you mean? | |
| 14:46:33 | sean-k-mooney | ya | |
| 14:46:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 14:46:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 14:46:51 | dansmith | well, the event coming before we wait for it is a pretty clear sign :) | |
| 14:46:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 14:46:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 14:46:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 14:46:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 14:47:25 | sean-k-mooney | dansmith: which event the neturon one? | |
| 14:47:36 | dansmith | sean-k-mooney: the plug event from neutron yeah | |
| 14:48:52 | sahid | it would be interesting to see the neutron logs | |
| 14:49:06 | sean-k-mooney | ya that will happen when the l2 agent polls the bridge config but im not sure that requires teh tap to exist, the fact that we have called brctl addif during plg might be enough for neutorn to "wire it up" and send the event | |
| 14:50:42 | sean-k-mooney | dansmith: nova/os-vif/libvirt handels l1 "phyical layer" neutron only wires up l2+ | |
| 14:52:16 | dansmith | sean-k-mooney: I'm not sure what your point is | |
| 14:53:06 | sean-k-mooney | dansmith: my point is im not sure the the neutron l2 agent ever check the tap actully exits vs just looking at the output of brctl show | |
| 14:53:29 | dansmith | sean-k-mooney: ah okay, well, seems like that has to be the case | |
| 14:53:59 | sean-k-mooney | not sure if that is a problem or not | |
| 14:54:28 | sean-k-mooney | when the tap shows up it should become part of the bridge correctly | |
| 14:54:32 | dansmith | well, tbh, getting the event before libvirt does the migration is really advantageous, because we can get our ducks lined up before we even start | |
| 14:54:54 | dansmith | because we don't need the sleep()-equivalent speed limit that way | |
| 14:55:03 | sean-k-mooney | im not sure if an TC rules for qos or iptables rules would be applied to the tap however but i have no evidnce either way | |
| 14:56:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 14:56:15 | sean-k-mooney | dansmith: ya makeing sure the network is configure so that it will jsut work when the tap shows up is good as it mean the RARP packets qemu sends before teh vm unpauses to set up mac learning will work | |
| 15:02:18 | sean-k-mooney | dansmith: do you have a link to matts change? | |
| 15:02:44 | mdbooth | sean-k-mooney: https://review.openstack.org/#/c/558001/10/nova/compute/manager.py I assume | |
| 15:02:51 | mdbooth | If you're still talking about the same thing. | |
| 15:03:12 | dansmith | sean-k-mooney: which one? | |
| 15:03:28 | sean-k-mooney | dansmith: moving the wait to compute manager | |
| 15:03:39 | dansmith | yeah pasted above, and mdbooth just pasted | |
| 15:03:43 | sean-k-mooney | i wanted to check the neutron logs for a message | |
| 15:04:28 | dansmith | sean-k-mooney: oh then you want the LB test patch | |
| 15:04:44 | sean-k-mooney | ya i think i found it | |
| 15:04:50 | sean-k-mooney | https://review.openstack.org/#/c/590585/ | |
| 15:05:20 | dansmith | sean-k-mooney: some good analysis in his comments on this: https://review.openstack.org/#/c/590538/ | |
| 15:07:37 | sean-k-mooney | dansmith: i wanted to check if this code was getting executed when plug was happening https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py#L529-L532 | |
| 15:08:14 | openstackgerrit | Merged openstack/nova master: add zvm into support matrix https://review.openstack.org/532720 | |
| 15:17:30 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Remove blacklisted py3 xen tests https://review.openstack.org/591419 | |
| 15:18:36 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Remove blacklisted py3 xen tests https://review.openstack.org/591419 | |
| 15:19:16 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable nested allocation candidates in scheduler https://review.openstack.org/585672 | |
| 15:19:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Functional test for booting with nested resources https://review.openstack.org/527728 | |
| 15:19:18 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Functional test for moving with nested resources https://review.openstack.org/587350 | |
| 15:19:42 | gibi | tetsuro: it went better than I expected ^^ | |
| 15:35:56 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 15:35:57 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 15:35:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 15:35:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 15:36:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 15:36:01 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 16:07:41 | dansmith | sean-k-mooney: sahid: as I said in the bug, it's easy for a loaded allinone gate worker to be slow enough that we win the race, | |
| 16:08:03 | dansmith | which I'm sure was happening when we originally tested it | |
| 16:08:04 | dansmith | if you look at mriedem's detailed analysis from last week, | |
| 16:08:09 | dansmith | he shows how he is able to poke it with the upstream jobs by tweaking the timing a bit | |
| 16:08:22 | dansmith | comments on PS2 here: https://review.openstack.org/#/c/590538/ | |
| 16:17:51 | sean-k-mooney | dansmith: right so the depeds on drops the time out to 30 seconds and truns on wait for plugs. the linux bridge live migration tests then fail. was this triggered sice the revert | |
| 16:24:15 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367 | |
| 16:25:15 | sean-k-mooney | im going to set up a lb env and test this locally. i want to test live migration betwen ovs and lb anyway so ill just check lb to lb first | |
| 16:40:36 | gryf | hi, can I get attention to this patch: https://review.openstack.org/#/c/536351/3 ? | |
| 16:45:47 | sean-k-mooney | gryf: looking now. | |
| 16:47:01 | gryf | sean-k-mooney, thank you | |
| 17:03:02 | openstackgerrit | Merged openstack/nova master: [placement] api-ref: Add missing aggregates example https://review.openstack.org/591332 | |
| 17:38:54 | sean-k-mooney | gryf: its not clear to me how the value is used | |
| 17:54:14 | efried | johnthetubaguy: --^ | |
| 19:14:34 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Remove blacklisted py3 xen tests https://review.openstack.org/591419 | |
| 19:45:26 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367 | |
| 20:14:42 | openstackgerrit | melanie witt proposed openstack/nova master: Add functional test for affinity with multiple cells https://review.openstack.org/585073 | |
| 20:14:43 | openstackgerrit | melanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258 | |
| 20:25:57 | openstackgerrit | Eric Fried proposed openstack/nova master: Improve debug logs for GET /allocation_candidates https://review.openstack.org/591510 | |