| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-02 | |||
| 22:09:08 | mnaser | And do and rdo to rdo migration and see which one of them is evaluating to false and not doing port bindings? | |
| 22:09:25 | mriedem | that code in post_live_migration_at_destination won't update the port binding host if it's already the specified host https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 | |
| 22:09:30 | mnaser | Cause rdo to rdo seems to do old school bindings from what I understand | |
| 22:09:32 | mriedem | which is why it's a no-op with the new flow | |
| 22:09:41 | mriedem | because we've already activated the dest host port binding before we get to https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 | |
| 22:10:46 | NewBruce | mnaser yeah, thats no problem (to update the conductor and test) | |
| 22:11:06 | sean-k-mooney | right be if we create an inactive binding on the dest and then somehow end ups runint the old code in post live migrate the host it gets back from neturon will be the source node as that will still be the active binding right | |
| 22:11:25 | mriedem | NewBruce: in case you're not familar, this is the code on the source compute that activates the dest port binding once we've switched to post-copy | |
| 22:11:25 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L1136-L1160 | |
| 22:11:47 | sean-k-mooney | so if we are corssing the streams here that check might be a little wierd | |
| 22:12:03 | NewBruce | mriedem right; yeah i have filled that with a ton of debug messages as well :D (have the logs saved down in my diaries if they become useful) | |
| 22:13:45 | mnaser | I think once we find out why rdo-rdo uses old flow even if upgrade levels are auto.. it might help | |
| 22:14:13 | cfriesen | when doing spec re-approvals for T for stuff that didn't make it in S, do I just copy from stein/approved into train/approved? | |
| 22:14:20 | mordred | mriedem: cool - thanks for the context on the migrate | |
| 22:15:09 | sean-k-mooney | mriedem: that code assumes we get the lifecycle event | |
| 22:15:15 | mriedem | cfriesen: https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications | |
| 22:15:20 | sean-k-mooney | if we dont we wont activate it. | |
| 22:15:49 | sean-k-mooney | until post live migration so may that could be whats happening in the RDO to OSA flow | |
| 22:15:50 | cfriesen | mriedem: you mean I'm supposed to read the readme? | |
| 22:16:05 | cfriesen | :) | |
| 22:16:05 | mriedem | sean-k-mooney: that code is just a best effort to try and activate the dest host port binding to reduce network downtime, as the comment says, " Otherwise the ports are bound to the destination host # in post_live_migration_at_destination." | |
| 22:16:45 | sean-k-mooney | ya but if we dont activate here then https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 will be true | |
| 22:17:13 | mnaser | not gonna lie, bugs like this makes me wish we had one agent that did it all :P | |
| 22:17:22 | mnaser | I’ll walk myself out | |
| 22:17:35 | mriedem | sean-k-mooney: sure, and honestly i thought that was still supposed to work | |
| 22:17:49 | mriedem | i don't know enough about what's happening within neutron to cause that duplicate primary key error | |
| 22:18:05 | sean-k-mooney | it proably shoudl work but i dont think we have testeded it. | |
| 22:18:35 | mriedem | the new flow deals with port bindings on the port bindings resources in the port bindings API, the old flow just deals with the port resource and its binding:host_id attribute | |
| 22:18:38 | NewBruce | i will post a large log dump into the launchpad… brb | |
| 22:18:42 | mnaser | mriedem: nova triés to update the old binding to point to the new host, but because a new binding already exists, it blows up | |
| 22:18:59 | mnaser | Cause you can’t have a binding with same port/host combo | |
| 22:19:00 | sean-k-mooney | we might be abel to reproduce the neutron issue with a functional test | |
| 22:19:11 | mriedem | mnaser: yeah, i just thought neutron was handling that for us | |
| 22:19:45 | mnaser | i guess maybe that’s where the bug lives | |
| 22:19:57 | mriedem | i.e. i thought if we changed the ports binding:host_id value and there was already a port binding for that host, but was inactive, neutron would automatically activate it | |
| 22:19:58 | sean-k-mooney | create a port binding, create an inactive port binding on another host, try to update the orginal binding to the destiation instead of activating | |
| 22:20:16 | sean-k-mooney | mriedem: i dont think it does that | |
| 22:20:29 | mnaser | Yep that would be a reproducer sean-k-mooney | |
| 22:20:37 | mriedem | sean-k-mooney: hell i could just push a patch to comment this out https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L1155 and our live migration job should blow up | |
| 22:21:56 | mriedem | NewBruce: while you're at it, dump the libvirtd and qemu-kvm package versions for both rdo and osa rocky nodes in the bu | |
| 22:21:58 | mriedem | *bug | |
| 22:22:17 | NewBruce | (i’ve just added to the ml2_port_binding trace to the launchpad as well as a debug version of the logs) | |
| 22:25:37 | openstackgerrit | Chris Friesen proposed openstack/nova-specs master: Re-propose emulated virtual TPM spec to train https://review.openstack.org/649463 | |
| 22:28:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Test theory about bug 1822884 https://review.openstack.org/649464 | |
| 22:28:07 | openstack | bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] https://launchpad.net/bugs/1822884 | |
| 22:28:07 | mriedem | mnaser: NewBruce: sean-k-mooney: ^ we'll find out | |
| 22:29:06 | mnaser | Let’s wait and see | |
| 22:29:41 | NewBruce | mriedem posted version info | |
| 22:30:32 | sean-k-mooney | i might be able to reporduce somethign on the neutron side too if i extend https://github.com/openstack/neutron/blob/master/neutron/tests/fullstack/test_ports_rebind.py but honestly im not sure that neutorn has mulit service functional test like nova has where we use a fake message bus and run everything in the one process | |
| 22:30:46 | NewBruce | mriedem will test it out | |
| 22:30:56 | mriedem | NewBruce: our ci system will test it | |
| 22:31:05 | mriedem | if that's the bug, the nova-live-migration job should explode | |
| 22:31:32 | mriedem | NewBruce: and it does look like your libvirt/qemu versions are different between rdo and osa so i wonder if that has something to do with the event getting sent or not | |
| 22:31:44 | mnaser | I’ll head off for a lil bit to doze and catch up on this buffer | |
| 22:32:44 | mriedem | reminds me that i still have https://review.openstack.org/#/c/594527/ | |
| 22:33:01 | mriedem | and https://review.openstack.org/#/c/594139/1 | |
| 22:33:31 | NewBruce | yep, id better get some shut eye soon too and check back in the morning. mnaser do you still want the debug from sean-k-mooney post earlier? | |
| 22:34:13 | NewBruce | mriedem any value in testing the service values? / disabling binding-extended | |
| 22:34:24 | NewBruce | ? | |
| 22:35:29 | mriedem | hmm, well we have a nova-grenade-live-migration job but would need to have that running on a stable/rocky change because then one compute would be queens and one would be rocky, or we could just pin rpc to queens in a job on master... | |
| 22:35:30 | sean-k-mooney | i dont think neutron provides a way to disabel binding extended via config so it would need a code chagne | |
| 22:35:39 | mriedem | sean-k-mooney: there is no api for that? | |
| 22:35:55 | mriedem | i guess not https://developer.openstack.org/api-ref/network/v2/index.html#id5 | |
| 22:35:59 | sean-k-mooney | i have looked before ill check but i dont think so | |
| 22:38:21 | NewBruce | i think we can run and upgrade accross all the compute nodes anyway; the issue we had with libevent seems to be isolated and easily tested for …. ok, lets see how that job comes out anyway | |
| 22:38:25 | NewBruce | cheers | |
| 22:38:55 | mriedem | posted https://review.openstack.org/649470 for the upgrade_levels/compute=queens pin simulation | |
| 22:39:14 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Update tests from fake_libvirt_util mocks https://review.openstack.org/649471 | |
| 22:41:30 | sean-k-mooney | mriedem: we expect this to show up in both the nova-live-migration and the grenade version right? | |
| 22:42:34 | sean-k-mooney | actully ill check it in the morning or ill fall a sleep watching devstack | |
| 22:42:37 | sean-k-mooney | night o/ | |
| 22:45:37 | openstackgerrit | Merged openstack/nova-specs master: Re-propose emulated virtual TPM spec to train https://review.openstack.org/649463 | |
| 23:19:04 | mriedem | ooooo hot dog i've got a devstack single node env with 2 non-cell0 cells, 2 computes on cell1 and 1 on cell2 | |
| 23:19:07 | mriedem | and it's pretty easy | |
| 23:41:53 | openstackgerrit | Merged openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376 | |
| 23:42:04 | openstackgerrit | Merged openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377 | |
| 23:42:13 | openstackgerrit | Merged openstack/nova master: Fix comment in test_attach_with_multiattach_fails_not_available https://review.openstack.org/649440 | |
| 23:47:48 | mriedem | heh just ran into bug 1781286 again | |
| 23:47:49 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,Triaged] https://launchpad.net/bugs/1781286 | |
| 23:47:54 | mriedem | we should maybe think about fixing that... | |
| 23:48:21 | mriedem | also, if things fail during server create rescheduling in conductor, chances are pretty good we don't set the instance to error status and it's stuck in build status | |
| #openstack-nova - 2019-04-03 | |||
| 00:16:52 | openstackgerrit | Merged openstack/nova master: Fix bug preventing forbidden traits from working https://review.openstack.org/648653 | |
| 00:17:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.openstack.org/641200 | |
| 00:18:41 | mriedem | oh wow this is pretty bad https://review.openstack.org/#/c/648653/ | |
| 00:19:09 | mriedem | efried: so forbidden traits just never worked in nova? ^ | |
| 01:13:21 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.openstack.org/648912 | |
| 01:13:21 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | |
| 03:04:58 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.openstack.org/648912 | |
| 03:04:59 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | |
| 06:39:48 | openstackgerrit | Merged openstack/nova master: Remove flavor id and name validation code https://review.openstack.org/638150 | |
| 07:44:10 | awalende | I want to set the numatune memory mode to "preferred" in nova, how do I do that in Queens? | |
| 08:06:00 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | |
| 08:11:02 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Re-propose the spec to allow specifying a list of CPU models https://review.openstack.org/642030 | |
| 08:16:30 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add node_uuid field to Destination object https://review.openstack.org/649532 | |
| 08:16:31 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Pass node uuid to new Destination.node_uuid https://review.openstack.org/649533 | |
| 08:16:32 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add in_tree field to RequestGroup object https://review.openstack.org/649534 | |
| 08:16:32 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: node_uuid from RequestSpec to ResourceRequest https://review.openstack.org/649535 | |
| 08:16:52 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment https://review.openstack.org/572790 | |
| 09:36:24 | openstackgerrit | Merged openstack/nova master: De-cruft compute manager live migration https://review.openstack.org/641449 | |
| 10:19:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unreachable codepaths https://review.openstack.org/649559 | |