| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-14 | |||
| 10:08:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 10:08:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 10:08:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 10:09:31 | sean-k-mooney | mriedem: yep i already had it open :) | |
| 10:21:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 10:26:28 | mriedem | logan-: did you get that resize bug opened? | |
| 10:44:02 | sean-k-mooney | looks like there was a good conversation in https://etherpad.openstack.org/p/BER-nfv-hpc-pain-points | |
| 11:13:08 | logan- | mriedem: https://bugs.launchpad.net/nova/+bug/1803331 | |
| 11:13:08 | openstack | Launchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New] | |
| 11:20:19 | lyarwood | logan-: https://review.openstack.org/#/c/578846/ might resolve that assuming this is a cause of us cleaning up aggresively after a failure, hard to tell from the trace alone. | |
| 11:22:53 | sean-k-mooney | logan-: how do you have an RBD backed flavor? isnt the storage backend determined by the nova.conf not the flaovor? | |
| 11:23:42 | lyarwood | sean-k-mooney: host aggregates | |
| 11:24:19 | sean-k-mooney | lyarwood: right but just because you are afinities flaovrs to host aggrages and that chnages your network backend does not mean this is related to flaovrs | |
| 11:24:23 | openstackgerrit | Sean McGinnis proposed openstack/nova master: Remove v1 check in Cinder client version lookup https://review.openstack.org/617927 | |
| 11:25:05 | sean-k-mooney | this is simply a bug with rezieing between image backed nodes and rbd nodes | |
| 11:25:08 | lyarwood | sean-k-mooney: imagebackend and yeah agreed I was just passing on what was described in the bug | |
| 11:25:30 | lyarwood | sean-k-mooney: this is all imagebackend, I'm not sure why logan- has called one out as imagebackend and the other as rbd | |
| 11:25:31 | mdbooth | lyarwood: reading | |
| 11:26:59 | sean-k-mooney | reading the bug it seams the issues is with migrating from imagebacked nodes to rbd backed nodes no? | |
| 11:27:01 | logan- | Sorry if it is confusing. Yes these are different nodes with different storage backends configured in nova.conf | |
| 11:27:14 | logan- | sean-k-mooney: correct | |
| 11:27:22 | sean-k-mooney | i would assume the same behavior would exist if you just migrated and did not reszie | |
| 11:27:25 | mdbooth | Sounds like there's 2 issues there | |
| 11:27:41 | mdbooth | Firstly, resizing between compute hosts using different imagebackends isn't supported | |
| 11:28:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 11:28:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 11:28:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 11:28:11 | mdbooth | So, I guess that's not an issue, except that we could do better at preventing the attempt | |
| 11:28:19 | mdbooth | I think sean-k-mooney mentioned that the other week. | |
| 11:28:25 | logan- | mdbooth: right, I don’t expect it to be. I just expect it to fail and not destroy the instance :) | |
| 11:28:35 | mdbooth | Secondly, we seem to have lost the root disk, which is extremely bad | |
| 11:28:58 | sean-k-mooney | logan-: i actully dont think its unresobaly to support we jsut dont today | |
| 11:29:47 | mdbooth | sean-k-mooney: Conceptually it's not unreasonable, it's just that the imagebackend code is utterly awful. | |
| 11:30:51 | sean-k-mooney | i was going to suggest makeing https://bugs.launchpad.net/nova/+bug/1803331 more general and simply stating "live migration between image backend types fails with loss of the root disk." | |
| 11:30:51 | openstack | Launchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New] | |
| 11:31:03 | mdbooth | sean-k-mooney: This is resize | |
| 11:31:15 | mdbooth | sean-k-mooney: Completely different code path. | |
| 11:31:19 | sean-k-mooney | mdbooth: yes it is but it would also break on migreate right | |
| 11:31:20 | mdbooth | Live migration might actually work. | |
| 11:31:25 | sean-k-mooney | sorry cold migrate | |
| 11:32:17 | sean-k-mooney | cold migrate and resize would both triger this. live migrate who knows | |
| 11:32:22 | mdbooth | logan-: At this point if you were a RH customer I'd ask you for sosreports. | |
| 11:32:54 | mdbooth | Basically I'd want to see full logs from the initial error which caused the root disk to be deleted. | |
| 11:33:32 | mdbooth | The subsequent error after the root disk has been deleted isn't all that interesting. | |
| 11:34:03 | sean-k-mooney | mdbooth: in theory this would alow apply to any missmatch in image backend right e.g. any combinantion lvm or image or rbd where the backend changes. | |
| 11:34:32 | mdbooth | sean-k-mooney: Yes. The code explicitly assumes that the layouts are identical on source and dest. | |
| 11:35:30 | sean-k-mooney | so this seams like something we should add to the host state object and then check in the scheuler in the future | |
| 11:35:44 | logan- | Ok I can pull more logs this afternoon. What would help? nova-compute in debug on both source & dest I’m guessing? | |
| 11:36:23 | mdbooth | logan-: Yep, specifically for the resize operation itself, not just the subsequent failure to boot | |
| 11:36:31 | logan- | Got it | |
| 11:36:35 | logan- | Will do. Thanks | |
| 11:40:41 | mdbooth | logan-: If you look in the instance directory on the destination you should find the root disk there | |
| 11:40:51 | mdbooth | logan-: Also, you should be able to revert resize | |
| 11:41:40 | mdbooth | I don't think we've actually deleted the root disk here, it's just that the dest is looking for it in the wrong place. | |
| 11:42:43 | sean-k-mooney | we might have deleted it on the source node | |
| 11:43:51 | sean-k-mooney | or do we not do that until we confirm and your guessing on revert it tries to find it using the location form the dest | |
| 11:45:09 | mdbooth | Not until confirm | |
| 11:45:20 | mdbooth | Revert will use the copy which remains on the source | |
| 11:46:29 | mdbooth | So actually I think this is just a case of extremely bad user experience. The operation isn't supported, but we could warn earlier rather than going down in flames. I don't believe any data has been lost. | |
| 11:47:03 | mdbooth | logan-: If ^^^ is correct, no need for those logs, btw. | |
| 11:56:42 | openstackgerrit | Merged openstack/nova stable/rocky: Set defult value of num_nvme_discover_tries=5 https://review.openstack.org/608683 | |
| 12:00:03 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Allow driver to specify switch&port for faster lookup https://review.openstack.org/617695 | |
| 12:52:40 | openstackgerrit | Merged openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | |
| 13:08:40 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:19:20 | logan- | updated the bug with additional details mdbooth | |
| 13:19:51 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 13:23:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 13:23:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 13:26:09 | mdbooth | logan-: Can you retry the revert after setting instance state to 'resize_confirm' ? | |
| 13:32:18 | ivaat | hi i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | |
| 13:32:18 | openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | |
| 13:42:59 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | |
| 13:47:25 | logan- | mdbooth: I guess that will require database shenanigans? "openstack server set: error: argument --state: invalid choice: u'resize_confirm' (choose from 'active', 'error')" | |
| 13:50:51 | mriedem | logan-: heh, reset state won't allow you to set it back to VERIFY_RESIZE | |
| 13:50:58 | mriedem | so yes, db surgery | |
| 13:51:12 | logan- | ok i'll give it a shot | |
| 13:51:39 | mriedem | because you need to get to this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3506 | |
| 13:53:56 | ivaat | hi! i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | |
| 13:53:56 | openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | |
| 13:54:04 | ivaat | i am sure it is bug | |
| 13:58:36 | mriedem | ivaat: see my comment in the bug, it's not a bug, unless you can investigate and prove otherwise | |
| 13:58:41 | mriedem | it looks like a misconfiguration | |
| 13:59:15 | ivaat | mriedem, well i attached nova.conf as well. tried user with logging in with same user. works | |
| 14:00:04 | ivaat | if it looks like misconfiguration then could you point out what part in config? i have gone trough config several times. followed official doc | |
| 14:00:38 | mriedem | see my comment on the bug | |
| 14:00:51 | ivaat | bug seems to be deleted | |
| 14:01:57 | ivaat | sry going directly with bug link i can see it now | |
| 14:02:49 | ivaat | mriedem, wtf. i just confirmed i have done this several times.. just take your time when helping. conf was attached to see what is wrong | |
| 14:03:36 | logan- | i tried hacking instances.vm_state='resized' and now it says "Instance has not been resized." so I guess there is some other state fields that need to be updated to hack it back to a revertable state | |
| 14:05:22 | mriedem | ivaat: i looked at the paste in the bug, i didn't see [keystone_authtoken] in there | |
| 14:07:31 | mriedem | logan-: so what did you do? vm was in error and you fixed the disk and tried to confirm again but that failed b/c it was in error rather than resized status? | |
| 14:07:37 | ivaat | at moment when i installed was no guide to add keystone_authtoken | |
| 14:07:55 | mriedem | ivaat: i don't know what install guide you were looking at | |
| 14:08:02 | mriedem | maybe something very old | |
| 14:09:12 | ivaat | alright, i will test it out | |
| 14:09:16 | ivaat | thx for now | |
| 14:10:15 | aspiers[m] | mriedem: does it make sense to aim for a quick hallway discussion about SEV while we're here? sean-k-mooney too if you're around | |
| 14:10:35 | ivaat | even nova conf is missing such keyword.. since it is section it should have been there | |