Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-13
16:48:42 sean-k-mooney stephenfin: do you recall if you wrote functional tests when you implemented the orgininal numa policies. if so ill extend them if not i guess ill write tests for both.
16:49:25 stephenfin I very much doubt it. I've only started writing those for the last two cycles or so
16:49:58 artom sean-k-mooney, I know I added https://review.opendev.org/#/c/682941/ recently
16:50:11 sean-k-mooney ok
16:50:22 sean-k-mooney artom: ya i rememebr
16:50:35 sean-k-mooney so those cover the alias based policies i think
16:50:38 sean-k-mooney you added the missing one
16:51:05 sean-k-mooney so i was more or less relying on thos but it hsould be easy to etend those to use the policy form the flavor
16:51:07 artom sean-k-mooney, well yeah, because we only have the alias based policies for now :)
16:52:03 sean-k-mooney ok ill push there version i have now after i adress you comments and ill start working on the func test after
16:54:04 openstackgerrit sean mooney proposed openstack/nova master: support pci numa affinity policies in flavor and image https://review.opendev.org/674072
17:09:10 openstackgerrit Eric Fried proposed openstack/nova master: Consolidate [image_cache] conf options https://review.opendev.org/690723
17:09:11 openstackgerrit Eric Fried proposed openstack/nova master: Add image caching to the support matrix https://review.opendev.org/690748
17:09:56 efried mriedem, stephenfin: Rebased and nit-fixed ^
17:12:15 openstackgerrit Stephen Finucane proposed openstack/nova master: "SUSPENDED" description changed in server_concepts guide and API REF https://review.opendev.org/663590
17:13:00 openstackgerrit Elod Illes proposed openstack/nova stable/pike: Error out interrupted builds https://review.opendev.org/687918
17:22:19 artom stephenfin, oh, friendly reminder that the NUMA LM func test stack is ready for another look: https://review.opendev.org/#/c/687404/
17:22:36 artom (I know it's not so much a learning curve as a learning brick wall)
17:24:10 stephenfin artom: Sure thing, but that's too much work for 5:20pm when jetlagged. I'll grab it in the morning :)
17:24:21 artom stephenfin, ack, thank you!
17:25:40 openstackgerrit Eric Fried proposed openstack/nova master: Remove functional test specific nova code https://review.opendev.org/683609
17:31:18 bauzas mriedem: dumb question but what the user sees from the API when we error out some instance because of NoValidHost ?
17:31:37 bauzas by a nova show I mean
17:32:00 bauzas mriedem: was about to say +1 to your ML thread, but was wondering the API result for the same
17:33:11 mriedem bauzas: the non-admin user? they see this:
17:33:11 mriedem $ openstack server show build-fail1 -f value -c fault {u'message': u'No valid host was found. ', u'code': 500, u'created': u'2019-11-13T15:57:13Z'}
17:33:27 mriedem the fault is only shown if the server status is ERROR or DELETED
17:33:46 bauzas mriedem: so the instance action event show should do the same
17:34:52 mriedem tbc, i wasn't proposing that the action event exception type is only shown for an ERROR or DELETED status server
17:35:08 mriedem since as i said in the thread, you can fail a resize and the server status doesn't go to ERROR
17:35:57 bauzas mriedem: yeah, I understood this
17:36:05 stephenfin efried: Think you could hold your nose on https://review.opendev.org/#/c/684345/16/nova/network/neutronv2/api.py to keep this moving, given I'm removing it again shortly after?
17:36:09 bauzas mriedem: that's why I think we should *also* do it
17:36:11 stephenfin drat, just missed him
17:36:37 bauzas and yeah, of course, when a user calls some ops, we can ask 'do a nova show'
17:36:58 bauzas but sometimes we need to look at the instance actions (like for resize) and that's why I'd love your spec :)
17:53:15 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345
17:53:15 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'os-security-group-default-rules' REST API https://review.opendev.org/686807
17:53:16 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused '*_default_rules' security group DB APIs https://review.opendev.org/686808
17:53:16 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove (most) '/os-networks' REST APIs https://review.opendev.org/686809
17:53:17 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove '/os-tenant-networks' REST API https://review.opendev.org/686810
17:53:17 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests https://review.opendev.org/686811
17:53:18 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812
17:53:18 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove nova-manage network, floating commands https://review.opendev.org/686813
17:53:19 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs https://review.opendev.org/686814
17:53:19 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary https://review.opendev.org/686815
17:53:20 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-network' binary https://review.opendev.org/686816
17:53:20 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Blast most references to nova-network https://review.opendev.org/686817
17:53:21 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP https://review.opendev.org/686818
20:08:05 openstackgerrit Dan Smith proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849
20:11:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Create instance action when burying in cell0 https://review.opendev.org/694165
20:27:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Create instance action when burying in cell0 https://review.opendev.org/694165
20:31:50 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: add troubleshooting guide for cleaning up orphaned allocations https://review.opendev.org/691427
21:41:02 openstackgerrit François Palin proposed openstack/nova master: Add retry to cinder api calls related to volume detach https://review.opendev.org/669674
21:44:59 openstackgerrit Merged openstack/nova master: Rename Claims resources to compute_node https://review.opendev.org/679470
21:45:06 openstackgerrit Merged openstack/nova master: Clear instance.launched_on when build fails https://review.opendev.org/683725
21:49:12 melwitt so... after helping a colleague unwedge a failed resize for a customer, I've learned that we intentionally don't roll back port bindings to the source when finish_resize fails https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5274
21:49:51 melwitt and the reasoning is because the assumption is "everything is fine" on the dest and that only the virt guest needs to be created
21:50:54 melwitt but, when finish_resize fails because update of volume attachments fails (error had to do with "duplicate connectors") it doesn't seem like the dest should really be considered an ok place for the instance to be
21:57:39 melwitt I wonder if fixing this might be as simple as moving the volume attach update before the network setup
21:57:46 melwitt on the dest
21:58:29 mriedem i feel like i recently added that comment but maybe not
21:58:56 mriedem we have a lot of "cleanup networking and cleanup volumes" in live and cold migrate/resize where none of it is really atomic
21:59:36 mriedem i made a note of that in the revert code for cross-cell resize here https://review.opendev.org/#/c/637647/51/nova/compute/manager.py@4690
22:00:19 mriedem i'm not sure what that duplicate connectors error is
22:00:55 mriedem ah that comment is relatively new https://review.opendev.org/#/c/635349/
22:01:22 melwitt yeah we're going to be investigating the root cause of the duplicate connectors thing, never seen that before
22:01:49 melwitt error came back from cinder
22:01:59 mriedem what was the volume type?
22:02:51 melwitt I dunno yet. we only focused on recovery for now, it's late at night for those who were fixing the wedged resize
22:03:40 mriedem so migrate_instance_finish updated the port bindings to point at the dest host and then _update_volume_attachments blew up right?
22:03:43 melwitt we (they) moved the port bindings back to the source manually and then we hard rebooted the instance to get it going for the customer again. they had only wanted to resize to a bigger flavor
22:03:47 melwitt yes
22:04:03 mriedem still, if you get to finish_resize the instance.host already points at the dest
22:04:12 mriedem so they would have had to do more than change port bindings
22:04:17 mriedem to get the guest to reboot on the source
22:04:32 mriedem resize_instance on the source updates the instance.host/node before casting to finish_resize on the dest
22:04:32 melwitt and it started up fine with the larger flavor once the port binding was put back on the source. and prior to that yes I had them change the host/node back to the source
22:04:38 melwitt yeah sorry
22:05:28 melwitt they'll have to fix the allocations too but the host is full so they're gonna move other stuff off and then fix once there's room
22:06:23 mriedem so even if we swapped migrate_instance_finish and _update_volume_attachments for this very specific fail case, the operator still has manual stuff they have to do, like updating the instance to point back at the source host
22:07:17 mriedem and if you do swap those, you likely should revert the instance host/node so hard rebooting on the source host works without manual db surgery
22:07:58 melwitt yeah, fair, but doing that is far easier than the port binding update. that was pretty involved and I personally didn't know how to do it. involved updating the virtual_interfaces table too I think
22:07:59 mriedem hell, even if finish_migration i'm not sure why we wouldn't roll everything back to the source host, but we've already updated port bindings and volume attachments by then which is likely why we say "you're on the dest now"
22:08:33 mriedem the virtual_interfaces table shouldn't have anything to do with it
22:08:40 mriedem unless they created new ports
22:09:05 melwitt ok, maybe they just looked for a uuid or something. I wasn't really understanding
22:10:31 melwitt looks like the volume type is "null" which would be the default
22:10:37 melwitt and I don't remember what the default is
22:10:40 mriedem well, if we changed the resize/cold migrate flow to use the multi-port binding api like we do for live migratoin since rocky thing would be a bit simpler to cleanup, because you'd have 2 port binding resources in neutron rather than one, one is active and one is inactive
22:11:04 mriedem but that's not a backportable change
22:11:18 mriedem and trying to rollback automatically from everything that could go wrong during finish_resize is likely to be pretty hairy
22:11:18 melwitt yeah, if port bindings AND volume attach succeed then saying you're on the dest now is fine
22:11:27 melwitt but if volumes fails, I don't see how dest could be ok
22:12:19 melwitt I was thinking of swapping the network setup and volume update only, in _finish_resize
22:13:44 melwitt so that if volume update fails, set the host/node back to the source like you said and bail. leave the port bindings alone
22:14:02 mriedem i don't think that gets you out of the woods,
22:14:31 mriedem because resize_instance on the source, before casting to finish_resize on the dest, deletes the old volume attachment with the source host connector and creates a new 'empty' volume attachment that gets updated on the dest with the dest host connector
22:14:37 mriedem see _terminate_volume_connections
22:14:44 melwitt O.o

Earlier   Later