| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-14 | |||
| 08:19:41 | dansmith | sean-k-mooney: boot or live migration with a target host | |
| 08:20:14 | sean-k-mooney | dansmith: the in_tree paramater? | |
| 08:20:41 | mriedem | correct | |
| 08:20:45 | mriedem | or rebuild with an image change | |
| 08:20:54 | mriedem | or evac with a target host | |
| 08:21:06 | mriedem | or cold migrate with a target host... :) | |
| 08:21:27 | sean-k-mooney | on the in_tree is set to the compute node uuid | |
| 08:21:36 | sean-k-mooney | ya that makes sense too | |
| 08:22:18 | openstackgerrit | Merged openstack/nova-specs master: Spec: allocation candidates in tree https://review.openstack.org/603585 | |
| 08:45:52 | mriedem | jroll: just some updates on https://review.openstack.org/#/c/609709/ and i'm happy | |
| 08:58:57 | openstackgerrit | sean mooney proposed openstack/os-vif master: always create ovs port during plug https://review.openstack.org/602384 | |
| 08:59:23 | openstackgerrit | sean mooney proposed openstack/os-vif master: add isolate_vif config option https://review.openstack.org/612534 | |
| 09:02:01 | openstackgerrit | sean mooney proposed openstack/os-vif stable/rocky: always create ovs port during plug https://review.openstack.org/609850 | |
| 09:31:42 | openstackgerrit | sean mooney proposed openstack/os-vif stable/rocky: add isolate_vif config option https://review.openstack.org/616285 | |
| 09:34:34 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: VMware: Attach volumes using adapter type from instance https://review.openstack.org/616599 | |
| 09:39:07 | jroll | mriedem: yep, doing it this morning | |
| 09:57:24 | mriedem | Kevin_Zheng: when you get a chance, can you update the root volume detach spec with the resize issue i told you about the other night? https://review.openstack.org/#/c/600628/ | |
| 09:58:31 | Kevin_Zheng | mriedem: Oh the spec, go it, I was only thinking about the implementation | |
| 10:04:42 | mriedem | i just want to make sure it's recorded in the spec so we don't forget about it during review | |
| 10:04:48 | mriedem | since the unshelve/start limitations are listed in the spec | |
| 10:05:00 | openstackgerrit | Jim Rollenhagen proposed openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | |
| 10:05:02 | mriedem | resize revert/confirm would have the same limitations - fail if there is no root volume | |
| 10:05:43 | jroll | mriedem: ^ should finally be good to go there now | |
| 10:07:23 | mriedem | ack | |
| 10:07:43 | mriedem | sean-k-mooney: if you want to follow along https://etherpad.openstack.org/p/BER-nfv-hpc-pain-points | |
| 10:08:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 10:08:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 10:08:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 10:08:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 10:09:31 | sean-k-mooney | mriedem: yep i already had it open :) | |
| 10:21:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 10:26:28 | mriedem | logan-: did you get that resize bug opened? | |
| 10:44:02 | sean-k-mooney | looks like there was a good conversation in https://etherpad.openstack.org/p/BER-nfv-hpc-pain-points | |
| 11:13:08 | openstack | Launchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New] | |
| 11:13:08 | logan- | mriedem: https://bugs.launchpad.net/nova/+bug/1803331 | |
| 11:20:19 | lyarwood | logan-: https://review.openstack.org/#/c/578846/ might resolve that assuming this is a cause of us cleaning up aggresively after a failure, hard to tell from the trace alone. | |
| 11:22:53 | sean-k-mooney | logan-: how do you have an RBD backed flavor? isnt the storage backend determined by the nova.conf not the flaovor? | |
| 11:23:42 | lyarwood | sean-k-mooney: host aggregates | |
| 11:24:19 | sean-k-mooney | lyarwood: right but just because you are afinities flaovrs to host aggrages and that chnages your network backend does not mean this is related to flaovrs | |
| 11:24:23 | openstackgerrit | Sean McGinnis proposed openstack/nova master: Remove v1 check in Cinder client version lookup https://review.openstack.org/617927 | |
| 11:25:05 | sean-k-mooney | this is simply a bug with rezieing between image backed nodes and rbd nodes | |
| 11:25:08 | lyarwood | sean-k-mooney: imagebackend and yeah agreed I was just passing on what was described in the bug | |
| 11:25:30 | lyarwood | sean-k-mooney: this is all imagebackend, I'm not sure why logan- has called one out as imagebackend and the other as rbd | |
| 11:25:31 | mdbooth | lyarwood: reading | |
| 11:26:59 | sean-k-mooney | reading the bug it seams the issues is with migrating from imagebacked nodes to rbd backed nodes no? | |
| 11:27:01 | logan- | Sorry if it is confusing. Yes these are different nodes with different storage backends configured in nova.conf | |
| 11:27:14 | logan- | sean-k-mooney: correct | |
| 11:27:22 | sean-k-mooney | i would assume the same behavior would exist if you just migrated and did not reszie | |
| 11:27:25 | mdbooth | Sounds like there's 2 issues there | |
| 11:27:41 | mdbooth | Firstly, resizing between compute hosts using different imagebackends isn't supported | |
| 11:28:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 11:28:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 11:28:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 11:28:11 | mdbooth | So, I guess that's not an issue, except that we could do better at preventing the attempt | |
| 11:28:19 | mdbooth | I think sean-k-mooney mentioned that the other week. | |
| 11:28:25 | logan- | mdbooth: right, I don’t expect it to be. I just expect it to fail and not destroy the instance :) | |
| 11:28:35 | mdbooth | Secondly, we seem to have lost the root disk, which is extremely bad | |
| 11:28:58 | sean-k-mooney | logan-: i actully dont think its unresobaly to support we jsut dont today | |
| 11:29:47 | mdbooth | sean-k-mooney: Conceptually it's not unreasonable, it's just that the imagebackend code is utterly awful. | |
| 11:30:51 | openstack | Launchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New] | |
| 11:30:51 | sean-k-mooney | i was going to suggest makeing https://bugs.launchpad.net/nova/+bug/1803331 more general and simply stating "live migration between image backend types fails with loss of the root disk." | |
| 11:31:03 | mdbooth | sean-k-mooney: This is resize | |
| 11:31:15 | mdbooth | sean-k-mooney: Completely different code path. | |
| 11:31:19 | sean-k-mooney | mdbooth: yes it is but it would also break on migreate right | |
| 11:31:20 | mdbooth | Live migration might actually work. | |
| 11:31:25 | sean-k-mooney | sorry cold migrate | |
| 11:32:17 | sean-k-mooney | cold migrate and resize would both triger this. live migrate who knows | |
| 11:32:22 | mdbooth | logan-: At this point if you were a RH customer I'd ask you for sosreports. | |
| 11:32:54 | mdbooth | Basically I'd want to see full logs from the initial error which caused the root disk to be deleted. | |
| 11:33:32 | mdbooth | The subsequent error after the root disk has been deleted isn't all that interesting. | |
| 11:34:03 | sean-k-mooney | mdbooth: in theory this would alow apply to any missmatch in image backend right e.g. any combinantion lvm or image or rbd where the backend changes. | |
| 11:34:32 | mdbooth | sean-k-mooney: Yes. The code explicitly assumes that the layouts are identical on source and dest. | |
| 11:35:30 | sean-k-mooney | so this seams like something we should add to the host state object and then check in the scheuler in the future | |
| 11:35:44 | logan- | Ok I can pull more logs this afternoon. What would help? nova-compute in debug on both source & dest I’m guessing? | |
| 11:36:23 | mdbooth | logan-: Yep, specifically for the resize operation itself, not just the subsequent failure to boot | |
| 11:36:31 | logan- | Got it | |
| 11:36:35 | logan- | Will do. Thanks | |
| 11:40:41 | mdbooth | logan-: If you look in the instance directory on the destination you should find the root disk there | |
| 11:40:51 | mdbooth | logan-: Also, you should be able to revert resize | |
| 11:41:40 | mdbooth | I don't think we've actually deleted the root disk here, it's just that the dest is looking for it in the wrong place. | |
| 11:42:43 | sean-k-mooney | we might have deleted it on the source node | |
| 11:43:51 | sean-k-mooney | or do we not do that until we confirm and your guessing on revert it tries to find it using the location form the dest | |
| 11:45:09 | mdbooth | Not until confirm | |
| 11:45:20 | mdbooth | Revert will use the copy which remains on the source | |
| 11:46:29 | mdbooth | So actually I think this is just a case of extremely bad user experience. The operation isn't supported, but we could warn earlier rather than going down in flames. I don't believe any data has been lost. | |
| 11:47:03 | mdbooth | logan-: If ^^^ is correct, no need for those logs, btw. | |
| 11:56:42 | openstackgerrit | Merged openstack/nova stable/rocky: Set defult value of num_nvme_discover_tries=5 https://review.openstack.org/608683 | |
| 12:00:03 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Allow driver to specify switch&port for faster lookup https://review.openstack.org/617695 | |
| 12:52:40 | openstackgerrit | Merged openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | |
| 13:08:40 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:19:20 | logan- | updated the bug with additional details mdbooth | |
| 13:19:51 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 13:26:09 | mdbooth | logan-: Can you retry the revert after setting instance state to 'resize_confirm' ? | |
| 13:32:18 | openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | |
| 13:32:18 | ivaat | hi i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | |
| 13:42:59 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |