Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-13
14:41:42 dansmith sahid: did you look at the bug reported by the guy on pike?
14:41:57 dansmith the event came *before* we even started listening for it
14:42:00 sean-k-mooney dansmith: for linux bridge no. we we do plug we call brctl add-interface bridge but the tap does not exist at that point
14:42:39 dansmith sean-k-mooney: do you mean add-br?
14:43:09 sean-k-mooney no ill get the link to the code in os-vif
14:43:16 gibi tetsuro: depending of the size of the confict I will do the rebase tomorrow
14:44:02 dansmith sean-k-mooney: sahid you should also look at matt's comments on the CI test patch where he follows the sequence of events in a CI run which was configured in such a way that we'd hit the timeout
14:44:06 sean-k-mooney actully wait for linux bridge plugin plug jsut creates teh bridge if it does not exist
14:44:32 dansmith sean-k-mooney: did you look at the bug where we can see the event coming *before* we start the live migration?
14:44:53 openstack Launchpad bug 1786346 in OpenStack Compute (nova) rocky "live migrations slow" [High,In progress] - Assigned to Matt Riedemann (mriedem)
14:44:53 dansmith https://bugs.launchpad.net/nova/+bug/1786346
14:45:07 dansmith and since that happens and we don't stop the migration,
14:45:18 dansmith the migration continues for a really long time at 1MB/s
14:45:27 tetsuro gibi: Sure... and feel free to ask me to do that. I'll check the IRC log, anyway.
14:45:28 dansmith the event did arrive, but before we were listening for it
14:45:34 dansmith so we failed unnecessarily,
14:45:41 dansmith and then the migration ran "forever"
14:46:09 sean-k-mooney actully if we pass the interface it does add it https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L149 its been a while since i looked at this code
14:46:11 gibi tetsuro: OK
14:46:26 dansmith sean-k-mooney: during plug you mean?
14:46:33 sean-k-mooney ya
14:46:50 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840
14:46:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267
14:46:51 dansmith well, the event coming before we wait for it is a pretty clear sign :)
14:46:52 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847
14:46:53 openstackgerrit Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268
14:46:54 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459
14:46:55 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
14:47:25 sean-k-mooney dansmith: which event the neturon one?
14:47:36 dansmith sean-k-mooney: the plug event from neutron yeah
14:48:52 sahid it would be interesting to see the neutron logs
14:49:06 sean-k-mooney ya that will happen when the l2 agent polls the bridge config but im not sure that requires teh tap to exist, the fact that we have called brctl addif during plg might be enough for neutorn to "wire it up" and send the event
14:50:42 sean-k-mooney dansmith: nova/os-vif/libvirt handels l1 "phyical layer" neutron only wires up l2+
14:52:16 dansmith sean-k-mooney: I'm not sure what your point is
14:53:06 sean-k-mooney dansmith: my point is im not sure the the neutron l2 agent ever check the tap actully exits vs just looking at the output of brctl show
14:53:29 dansmith sean-k-mooney: ah okay, well, seems like that has to be the case
14:53:59 sean-k-mooney not sure if that is a problem or not
14:54:28 sean-k-mooney when the tap shows up it should become part of the bridge correctly
14:54:32 dansmith well, tbh, getting the event before libvirt does the migration is really advantageous, because we can get our ducks lined up before we even start
14:54:54 dansmith because we don't need the sleep()-equivalent speed limit that way
14:55:03 sean-k-mooney im not sure if an TC rules for qos or iptables rules would be applied to the tap however but i have no evidnce either way
14:56:07 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667
14:56:15 sean-k-mooney dansmith: ya makeing sure the network is configure so that it will jsut work when the tap shows up is good as it mean the RARP packets qemu sends before teh vm unpauses to set up mac learning will work
15:02:18 sean-k-mooney dansmith: do you have a link to matts change?
15:02:44 mdbooth sean-k-mooney: https://review.openstack.org/#/c/558001/10/nova/compute/manager.py I assume
15:02:51 mdbooth If you're still talking about the same thing.
15:03:12 dansmith sean-k-mooney: which one?
15:03:28 sean-k-mooney dansmith: moving the wait to compute manager
15:03:39 dansmith yeah pasted above, and mdbooth just pasted
15:03:43 sean-k-mooney i wanted to check the neutron logs for a message
15:04:28 dansmith sean-k-mooney: oh then you want the LB test patch
15:04:44 sean-k-mooney ya i think i found it
15:04:50 sean-k-mooney https://review.openstack.org/#/c/590585/
15:05:20 dansmith sean-k-mooney: some good analysis in his comments on this: https://review.openstack.org/#/c/590538/
15:07:37 sean-k-mooney dansmith: i wanted to check if this code was getting executed when plug was happening https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py#L529-L532
15:08:14 openstackgerrit Merged openstack/nova master: add zvm into support matrix https://review.openstack.org/532720
15:17:30 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Remove blacklisted py3 xen tests https://review.openstack.org/591419
15:18:36 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Remove blacklisted py3 xen tests https://review.openstack.org/591419
15:19:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable nested allocation candidates in scheduler https://review.openstack.org/585672
15:19:17 openstackgerrit Balazs Gibizer proposed openstack/nova master: Functional test for booting with nested resources https://review.openstack.org/527728
15:19:18 openstackgerrit Balazs Gibizer proposed openstack/nova master: Functional test for moving with nested resources https://review.openstack.org/587350
15:19:42 gibi tetsuro: it went better than I expected ^^
15:35:56 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840
15:35:57 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267
15:35:58 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847
15:35:59 openstackgerrit Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268
15:36:00 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459
15:36:01 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
16:07:41 dansmith sean-k-mooney: sahid: as I said in the bug, it's easy for a loaded allinone gate worker to be slow enough that we win the race,
16:08:03 dansmith which I'm sure was happening when we originally tested it
16:08:04 dansmith if you look at mriedem's detailed analysis from last week,
16:08:09 dansmith he shows how he is able to poke it with the upstream jobs by tweaking the timing a bit
16:08:22 dansmith comments on PS2 here: https://review.openstack.org/#/c/590538/
16:17:51 sean-k-mooney dansmith: right so the depeds on drops the time out to 30 seconds and truns on wait for plugs. the linux bridge live migration tests then fail. was this triggered sice the revert
16:24:15 openstackgerrit Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367
16:25:15 sean-k-mooney im going to set up a lb env and test this locally. i want to test live migration betwen ovs and lb anyway so ill just check lb to lb first
16:40:36 gryf hi, can I get attention to this patch: https://review.openstack.org/#/c/536351/3 ?
16:45:47 sean-k-mooney gryf: looking now.
16:47:01 gryf sean-k-mooney, thank you
17:03:02 openstackgerrit Merged openstack/nova master: [placement] api-ref: Add missing aggregates example https://review.openstack.org/591332
17:38:54 sean-k-mooney gryf: its not clear to me how the value is used
17:54:14 efried johnthetubaguy: --^
19:14:34 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Remove blacklisted py3 xen tests https://review.openstack.org/591419
19:45:26 openstackgerrit Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367
20:14:42 openstackgerrit melanie witt proposed openstack/nova master: Add functional test for affinity with multiple cells https://review.openstack.org/585073
20:14:43 openstackgerrit melanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258
20:25:57 openstackgerrit Eric Fried proposed openstack/nova master: Improve debug logs for GET /allocation_candidates https://review.openstack.org/591510
#openstack-nova - 2018-08-14
00:33:51 mriedem o/
00:40:39 Kevin_Zheng too early
00:45:30 Kevin_Zheng mriedem, got something interesting, as we can provide metadata with this API, https://developer.openstack.org/api-ref/compute/#create-image-createimage-action it might override the system generated metadata: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2877
00:45:58 Kevin_Zheng and we did it again in https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2911
00:46:19 Kevin_Zheng seems redundant, and should we limit the key?
00:46:45 Kevin_Zheng like don't allow set user_id
00:46:58 Kevin_Zheng image_type
00:47:00 mriedem my network needs to catch up...
00:47:28 Kevin_Zheng OK
00:55:22 mriedem Kevin_Zheng: hmm, yeah, not sure
00:55:42 mriedem looks like this could be used to change the owner of the snapshot to another user
00:56:51 Kevin_Zheng yeah
00:57:18 Kevin_Zheng Yeah

Earlier   Later