| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-13 | |||
| 10:00:25 | mriedem | alright, going to dinner | |
| 10:00:39 | Kevin_Zheng | Coll | |
| 10:00:41 | Kevin_Zheng | Cool | |
| 10:08:11 | mdbooth | ratailor: I wonder if we can get zzzeek to comment on http://lists.openstack.org/pipermail/openstack-dev/2018-August/133202.html | |
| 10:11:20 | ratailor | mdbooth, Thanks for putting it on ML nicely, I think we should get comment from him. | |
| 10:12:14 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] api-ref: Add missing aggregates example https://review.openstack.org/591332 | |
| 10:25:46 | openstackgerrit | Rajesh Tailor proposed openstack/nova master: Fix case-sensitivity for metadata keys https://review.openstack.org/504885 | |
| 10:38:03 | nalhat | Hi, Is any one aware about the filters used by nova: 'AggregateInstanceExtraSpecsFilter' while instance reservation(Blazar) | |
| 10:40:43 | nalhat | I was referring this doc: https://docs.openstack.org/blazar/latest/cli/instance-reservation.html for instance reservation by Blazar, at the time of creating an instance in nova with reservation id this filter comes in picture | |
| 11:52:07 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_compute_api.py (4) https://review.openstack.org/568462 | |
| 11:54:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: VMware: fix TypeError while get console log https://review.openstack.org/591365 | |
| 12:00:12 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153 | |
| 12:05:15 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (5) https://review.openstack.org/570842 | |
| 12:10:05 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (6) https://review.openstack.org/571330 | |
| 12:13:18 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367 | |
| 12:14:46 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992 | |
| 12:18:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993 | |
| 12:22:25 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 13:10:14 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] api-ref: Add missing aggregates example https://review.openstack.org/591332 | |
| 13:47:46 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367 | |
| 13:50:22 | efried | Scheduler meeting in 10 minutes in #openstack-meeting-alt | |
| 14:30:38 | dansmith | sahid: we wanted to make sure you saw this: https://review.openstack.org/#/c/591275/ | |
| 14:30:54 | dansmith | sahid: and that the rationale was clear, because we discussed mostly on irc | |
| 14:31:28 | dansmith | basically, I think that doing that in the libvirt driver will always be racy because of where the plug happens, | |
| 14:32:03 | dansmith | but since mriedem added it generically in the compute manager (where it can be done properly) it has the same effect you were going for | |
| 14:32:22 | dansmith | and, the two conflict of course because the wait in compute manager will "eat" the notification so that the libvirt wait will never see it | |
| 14:33:13 | dansmith | sahid: I also wonder if the race we identified is why you were seeing it not always work sometimes | |
| 14:33:30 | dansmith | the person that reported it on pike was always seeing it fail to win the race and thus never increase the speed limit | |
| 14:33:40 | gibi | tetsuro: would you mind if I rebase you report client 1.29 patches? | |
| 14:34:10 | gibi | tetsuro: I would like to see if 1.28 and 1.29 are still compatible codewise before I move forward with the split | |
| 14:34:14 | sahid | dansmith: initially i proposed to continue live-migration is a timeout occurs, since we know that some network issues might happen wich does not necessary mean the live-migration is failling | |
| 14:34:28 | tetsuro | gibi: no I don't mind at all. thanks! | |
| 14:35:00 | dansmith | sahid: right, which I'm not okay with, but I wonder if the "network issues" you saw were actually just because your approach was racy | |
| 14:35:33 | dansmith | sahid: here's the patch that added it to compute manager: https://review.openstack.org/#/c/558001/10/nova/compute/manager.py | |
| 14:35:53 | dansmith | sahid: which starts the wait *before* the pre-live-on-dest call runs which actually does the plug, so it's not racy | |
| 14:36:33 | dansmith | and you can see, that patch will consume the notification so that your wait in libvirt would never see it | |
| 14:36:52 | sean-k-mooney | dansmith: sahid well reducing the live migration bandwith was really the equvalent of a random sleep(x) as it was never guarneteed to be enough as even with the minium migration bandwith i could still complete before the netwroking was set up | |
| 14:37:34 | dansmith | sean-k-mooney: yeah I said that several times on the initial review :) | |
| 14:39:22 | dansmith | the assertion was made that we could only do it where he had it because of when/where the plug happens, | |
| 14:39:32 | dansmith | and that slowing the migration was the only way to catch it first, | |
| 14:39:38 | dansmith | but that clearly wasn't correct | |
| 14:39:57 | dansmith | so matt's version in compute manager does the full wait before we even start, which is definitely better | |
| 14:40:44 | sahid | how that is supposed to work? linux bridge agent is configuring the network only when it see the new tap device created | |
| 14:41:17 | sahid | that is basically why i took that way to fix the issue, libvirt is creating the tap device when calling migrateToURI | |
| 14:41:18 | dansmith | right, but that happens when we plug the vif, yeah? | |
| 14:41:27 | sahid | no | |
| 14:41:42 | dansmith | sahid: did you look at the bug reported by the guy on pike? | |
| 14:41:57 | dansmith | the event came *before* we even started listening for it | |
| 14:42:00 | sean-k-mooney | dansmith: for linux bridge no. we we do plug we call brctl add-interface bridge but the tap does not exist at that point | |
| 14:42:39 | dansmith | sean-k-mooney: do you mean add-br? | |
| 14:43:09 | sean-k-mooney | no ill get the link to the code in os-vif | |
| 14:43:16 | gibi | tetsuro: depending of the size of the confict I will do the rebase tomorrow | |
| 14:44:02 | dansmith | sean-k-mooney: sahid you should also look at matt's comments on the CI test patch where he follows the sequence of events in a CI run which was configured in such a way that we'd hit the timeout | |
| 14:44:06 | sean-k-mooney | actully wait for linux bridge plugin plug jsut creates teh bridge if it does not exist | |
| 14:44:32 | dansmith | sean-k-mooney: did you look at the bug where we can see the event coming *before* we start the live migration? | |
| 14:44:53 | openstack | Launchpad bug 1786346 in OpenStack Compute (nova) rocky "live migrations slow" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 14:44:53 | dansmith | https://bugs.launchpad.net/nova/+bug/1786346 | |
| 14:45:07 | dansmith | and since that happens and we don't stop the migration, | |
| 14:45:18 | dansmith | the migration continues for a really long time at 1MB/s | |
| 14:45:27 | tetsuro | gibi: Sure... and feel free to ask me to do that. I'll check the IRC log, anyway. | |
| 14:45:28 | dansmith | the event did arrive, but before we were listening for it | |
| 14:45:34 | dansmith | so we failed unnecessarily, | |
| 14:45:41 | dansmith | and then the migration ran "forever" | |
| 14:46:09 | sean-k-mooney | actully if we pass the interface it does add it https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L149 its been a while since i looked at this code | |
| 14:46:11 | gibi | tetsuro: OK | |
| 14:46:26 | dansmith | sean-k-mooney: during plug you mean? | |
| 14:46:33 | sean-k-mooney | ya | |
| 14:46:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 14:46:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 14:46:51 | dansmith | well, the event coming before we wait for it is a pretty clear sign :) | |
| 14:46:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 14:46:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 14:46:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 14:46:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 14:47:25 | sean-k-mooney | dansmith: which event the neturon one? | |
| 14:47:36 | dansmith | sean-k-mooney: the plug event from neutron yeah | |
| 14:48:52 | sahid | it would be interesting to see the neutron logs | |
| 14:49:06 | sean-k-mooney | ya that will happen when the l2 agent polls the bridge config but im not sure that requires teh tap to exist, the fact that we have called brctl addif during plg might be enough for neutorn to "wire it up" and send the event | |
| 14:50:42 | sean-k-mooney | dansmith: nova/os-vif/libvirt handels l1 "phyical layer" neutron only wires up l2+ | |
| 14:52:16 | dansmith | sean-k-mooney: I'm not sure what your point is | |
| 14:53:06 | sean-k-mooney | dansmith: my point is im not sure the the neutron l2 agent ever check the tap actully exits vs just looking at the output of brctl show | |
| 14:53:29 | dansmith | sean-k-mooney: ah okay, well, seems like that has to be the case | |
| 14:53:59 | sean-k-mooney | not sure if that is a problem or not | |
| 14:54:28 | sean-k-mooney | when the tap shows up it should become part of the bridge correctly | |
| 14:54:32 | dansmith | well, tbh, getting the event before libvirt does the migration is really advantageous, because we can get our ducks lined up before we even start | |
| 14:54:54 | dansmith | because we don't need the sleep()-equivalent speed limit that way | |
| 14:55:03 | sean-k-mooney | im not sure if an TC rules for qos or iptables rules would be applied to the tap however but i have no evidnce either way | |
| 14:56:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 14:56:15 | sean-k-mooney | dansmith: ya makeing sure the network is configure so that it will jsut work when the tap shows up is good as it mean the RARP packets qemu sends before teh vm unpauses to set up mac learning will work | |
| 15:02:18 | sean-k-mooney | dansmith: do you have a link to matts change? | |
| 15:02:44 | mdbooth | sean-k-mooney: https://review.openstack.org/#/c/558001/10/nova/compute/manager.py I assume | |
| 15:02:51 | mdbooth | If you're still talking about the same thing. | |
| 15:03:12 | dansmith | sean-k-mooney: which one? | |
| 15:03:28 | sean-k-mooney | dansmith: moving the wait to compute manager | |
| 15:03:39 | dansmith | yeah pasted above, and mdbooth just pasted | |
| 15:03:43 | sean-k-mooney | i wanted to check the neutron logs for a message | |
| 15:04:28 | dansmith | sean-k-mooney: oh then you want the LB test patch | |
| 15:04:44 | sean-k-mooney | ya i think i found it | |
| 15:04:50 | sean-k-mooney | https://review.openstack.org/#/c/590585/ | |
| 15:05:20 | dansmith | sean-k-mooney: some good analysis in his comments on this: https://review.openstack.org/#/c/590538/ | |