| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-02 | |||
| 22:11:25 | mriedem | NewBruce: in case you're not familar, this is the code on the source compute that activates the dest port binding once we've switched to post-copy | |
| 22:11:25 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L1136-L1160 | |
| 22:11:47 | sean-k-mooney | so if we are corssing the streams here that check might be a little wierd | |
| 22:12:03 | NewBruce | mriedem right; yeah i have filled that with a ton of debug messages as well :D (have the logs saved down in my diaries if they become useful) | |
| 22:13:45 | mnaser | I think once we find out why rdo-rdo uses old flow even if upgrade levels are auto.. it might help | |
| 22:14:13 | cfriesen | when doing spec re-approvals for T for stuff that didn't make it in S, do I just copy from stein/approved into train/approved? | |
| 22:14:20 | mordred | mriedem: cool - thanks for the context on the migrate | |
| 22:15:09 | sean-k-mooney | mriedem: that code assumes we get the lifecycle event | |
| 22:15:15 | mriedem | cfriesen: https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications | |
| 22:15:20 | sean-k-mooney | if we dont we wont activate it. | |
| 22:15:49 | sean-k-mooney | until post live migration so may that could be whats happening in the RDO to OSA flow | |
| 22:15:50 | cfriesen | mriedem: you mean I'm supposed to read the readme? | |
| 22:16:05 | cfriesen | :) | |
| 22:16:05 | mriedem | sean-k-mooney: that code is just a best effort to try and activate the dest host port binding to reduce network downtime, as the comment says, " Otherwise the ports are bound to the destination host # in post_live_migration_at_destination." | |
| 22:16:45 | sean-k-mooney | ya but if we dont activate here then https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 will be true | |
| 22:17:13 | mnaser | not gonna lie, bugs like this makes me wish we had one agent that did it all :P | |
| 22:17:22 | mnaser | I’ll walk myself out | |
| 22:17:35 | mriedem | sean-k-mooney: sure, and honestly i thought that was still supposed to work | |
| 22:17:49 | mriedem | i don't know enough about what's happening within neutron to cause that duplicate primary key error | |
| 22:18:05 | sean-k-mooney | it proably shoudl work but i dont think we have testeded it. | |
| 22:18:35 | mriedem | the new flow deals with port bindings on the port bindings resources in the port bindings API, the old flow just deals with the port resource and its binding:host_id attribute | |
| 22:18:38 | NewBruce | i will post a large log dump into the launchpad… brb | |
| 22:18:42 | mnaser | mriedem: nova triés to update the old binding to point to the new host, but because a new binding already exists, it blows up | |
| 22:18:59 | mnaser | Cause you can’t have a binding with same port/host combo | |
| 22:19:00 | sean-k-mooney | we might be abel to reproduce the neutron issue with a functional test | |
| 22:19:11 | mriedem | mnaser: yeah, i just thought neutron was handling that for us | |
| 22:19:45 | mnaser | i guess maybe that’s where the bug lives | |
| 22:19:57 | mriedem | i.e. i thought if we changed the ports binding:host_id value and there was already a port binding for that host, but was inactive, neutron would automatically activate it | |
| 22:19:58 | sean-k-mooney | create a port binding, create an inactive port binding on another host, try to update the orginal binding to the destiation instead of activating | |
| 22:20:16 | sean-k-mooney | mriedem: i dont think it does that | |
| 22:20:29 | mnaser | Yep that would be a reproducer sean-k-mooney | |
| 22:20:37 | mriedem | sean-k-mooney: hell i could just push a patch to comment this out https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L1155 and our live migration job should blow up | |
| 22:21:56 | mriedem | NewBruce: while you're at it, dump the libvirtd and qemu-kvm package versions for both rdo and osa rocky nodes in the bu | |
| 22:21:58 | mriedem | *bug | |
| 22:22:17 | NewBruce | (i’ve just added to the ml2_port_binding trace to the launchpad as well as a debug version of the logs) | |
| 22:25:37 | openstackgerrit | Chris Friesen proposed openstack/nova-specs master: Re-propose emulated virtual TPM spec to train https://review.openstack.org/649463 | |
| 22:28:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Test theory about bug 1822884 https://review.openstack.org/649464 | |
| 22:28:07 | openstack | bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] https://launchpad.net/bugs/1822884 | |
| 22:28:07 | mriedem | mnaser: NewBruce: sean-k-mooney: ^ we'll find out | |
| 22:29:06 | mnaser | Let’s wait and see | |
| 22:29:41 | NewBruce | mriedem posted version info | |
| 22:30:32 | sean-k-mooney | i might be able to reporduce somethign on the neutron side too if i extend https://github.com/openstack/neutron/blob/master/neutron/tests/fullstack/test_ports_rebind.py but honestly im not sure that neutorn has mulit service functional test like nova has where we use a fake message bus and run everything in the one process | |
| 22:30:46 | NewBruce | mriedem will test it out | |
| 22:30:56 | mriedem | NewBruce: our ci system will test it | |
| 22:31:05 | mriedem | if that's the bug, the nova-live-migration job should explode | |
| 22:31:32 | mriedem | NewBruce: and it does look like your libvirt/qemu versions are different between rdo and osa so i wonder if that has something to do with the event getting sent or not | |
| 22:31:44 | mnaser | I’ll head off for a lil bit to doze and catch up on this buffer | |
| 22:32:44 | mriedem | reminds me that i still have https://review.openstack.org/#/c/594527/ | |
| 22:33:01 | mriedem | and https://review.openstack.org/#/c/594139/1 | |
| 22:33:31 | NewBruce | yep, id better get some shut eye soon too and check back in the morning. mnaser do you still want the debug from sean-k-mooney post earlier? | |
| 22:34:13 | NewBruce | mriedem any value in testing the service values? / disabling binding-extended | |
| 22:34:24 | NewBruce | ? | |
| 22:35:29 | mriedem | hmm, well we have a nova-grenade-live-migration job but would need to have that running on a stable/rocky change because then one compute would be queens and one would be rocky, or we could just pin rpc to queens in a job on master... | |
| 22:35:30 | sean-k-mooney | i dont think neutron provides a way to disabel binding extended via config so it would need a code chagne | |
| 22:35:39 | mriedem | sean-k-mooney: there is no api for that? | |
| 22:35:55 | mriedem | i guess not https://developer.openstack.org/api-ref/network/v2/index.html#id5 | |
| 22:35:59 | sean-k-mooney | i have looked before ill check but i dont think so | |
| 22:38:21 | NewBruce | i think we can run and upgrade accross all the compute nodes anyway; the issue we had with libevent seems to be isolated and easily tested for …. ok, lets see how that job comes out anyway | |
| 22:38:25 | NewBruce | cheers | |
| 22:38:55 | mriedem | posted https://review.openstack.org/649470 for the upgrade_levels/compute=queens pin simulation | |
| 22:39:14 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Update tests from fake_libvirt_util mocks https://review.openstack.org/649471 | |
| 22:41:30 | sean-k-mooney | mriedem: we expect this to show up in both the nova-live-migration and the grenade version right? | |
| 22:42:34 | sean-k-mooney | actully ill check it in the morning or ill fall a sleep watching devstack | |
| 22:42:37 | sean-k-mooney | night o/ | |
| 22:45:37 | openstackgerrit | Merged openstack/nova-specs master: Re-propose emulated virtual TPM spec to train https://review.openstack.org/649463 | |
| 23:19:04 | mriedem | ooooo hot dog i've got a devstack single node env with 2 non-cell0 cells, 2 computes on cell1 and 1 on cell2 | |
| 23:19:07 | mriedem | and it's pretty easy | |
| 23:41:53 | openstackgerrit | Merged openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376 | |
| 23:42:04 | openstackgerrit | Merged openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377 | |
| 23:42:13 | openstackgerrit | Merged openstack/nova master: Fix comment in test_attach_with_multiattach_fails_not_available https://review.openstack.org/649440 | |
| 23:47:48 | mriedem | heh just ran into bug 1781286 again | |
| 23:47:49 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,Triaged] https://launchpad.net/bugs/1781286 | |
| 23:47:54 | mriedem | we should maybe think about fixing that... | |
| 23:48:21 | mriedem | also, if things fail during server create rescheduling in conductor, chances are pretty good we don't set the instance to error status and it's stuck in build status | |
| #openstack-nova - 2019-04-03 | |||
| 00:16:52 | openstackgerrit | Merged openstack/nova master: Fix bug preventing forbidden traits from working https://review.openstack.org/648653 | |
| 00:17:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.openstack.org/641200 | |
| 00:18:41 | mriedem | oh wow this is pretty bad https://review.openstack.org/#/c/648653/ | |
| 00:19:09 | mriedem | efried: so forbidden traits just never worked in nova? ^ | |
| 01:13:21 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.openstack.org/648912 | |
| 01:13:21 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | |
| 03:04:58 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.openstack.org/648912 | |
| 03:04:59 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | |
| 06:39:48 | openstackgerrit | Merged openstack/nova master: Remove flavor id and name validation code https://review.openstack.org/638150 | |
| 07:44:10 | awalende | I want to set the numatune memory mode to "preferred" in nova, how do I do that in Queens? | |
| 08:06:00 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | |
| 08:11:02 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Re-propose the spec to allow specifying a list of CPU models https://review.openstack.org/642030 | |
| 08:16:30 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add node_uuid field to Destination object https://review.openstack.org/649532 | |
| 08:16:31 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Pass node uuid to new Destination.node_uuid https://review.openstack.org/649533 | |
| 08:16:32 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add in_tree field to RequestGroup object https://review.openstack.org/649534 | |
| 08:16:32 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: node_uuid from RequestSpec to ResourceRequest https://review.openstack.org/649535 | |
| 08:16:52 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Block swap volume on volumes with >1 rw attachment https://review.openstack.org/572790 | |
| 09:36:24 | openstackgerrit | Merged openstack/nova master: De-cruft compute manager live migration https://review.openstack.org/641449 | |
| 10:19:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unreachable codepaths https://review.openstack.org/649559 | |
| 10:27:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Group API versions by release https://review.openstack.org/649560 | |
| 10:27:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Trivial fixes to API version history https://review.openstack.org/649561 | |
| 10:27:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove dead code https://review.openstack.org/649562 | |
| 10:27:56 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hacking: Fix dodgy check https://review.openstack.org/649563 | |
| 10:27:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: zvm: Remove dead code https://review.openstack.org/649564 | |
| 10:27:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove dead 'ALIAS' constant https://review.openstack.org/649565 | |
| 10:27:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove dead placement API functions https://review.openstack.org/649566 | |