Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-13
16:17:13 mriedem dansmith: ah ok
16:17:25 mriedem i'll open a bug in a bit
16:20:51 sean-k-mooney am i can if you would like them
16:21:23 mriedem ?
16:21:23 mriedem anything involving image meta / flavor extra specs + pci + affinity + scheduling + compute likely means unit tests aren't sufficient, yeah
16:21:29 mriedem maybe that's just me
16:21:48 openstackgerrit Alexandre arents proposed openstack/nova master: Abort live-migration during instance_init https://review.opendev.org/678016
16:23:26 sean-k-mooney mriedem: the code change is just reuseing the existing support for numa policies
16:23:44 sean-k-mooney so im jsut passing that policy via the flavor instead of the alisa
16:24:00 sean-k-mooney that said we may not have existing fucntional test for that so i will look
16:27:57 sean-k-mooney mriedem: i was kind of assuming the existing functional test for the numa polices would be sufficent but i will check.
16:30:06 mriedem i never assume anything relating to numa test coverage is sufficient for new code dealing with numa
16:30:17 sean-k-mooney fair
16:30:22 mriedem but i'm a cranky old troll
16:37:21 openstack bug 1833581 in OpenStack Compute (nova) train "instance stuck in BUILD state if nova-compute is restarted" [Low,Fix committed] https://launchpad.net/bugs/1833581 - Assigned to Balazs Gibizer (balazs-gibizer)
16:37:21 openstackgerrit Elod Illes proposed openstack/nova stable/pike: Functional reproduce for bug 1833581 https://review.opendev.org/687917
16:38:13 mriedem melwitt: can you hit https://review.opendev.org/#/c/693554/ to keep amodi's doc bug fix backports moving?
16:45:02 melwitt mriedem: yup, thanks
16:45:10 stephenfin mriedem: You're not that old
16:45:30 stephenfin 🙃
16:47:45 sean-k-mooney artom: looks like i have already fixed most of your comments locally im going to fix the last few then ill push it up.
16:48:42 sean-k-mooney stephenfin: do you recall if you wrote functional tests when you implemented the orgininal numa policies. if so ill extend them if not i guess ill write tests for both.
16:49:25 stephenfin I very much doubt it. I've only started writing those for the last two cycles or so
16:49:58 artom sean-k-mooney, I know I added https://review.opendev.org/#/c/682941/ recently
16:50:11 sean-k-mooney ok
16:50:22 sean-k-mooney artom: ya i rememebr
16:50:35 sean-k-mooney so those cover the alias based policies i think
16:50:38 sean-k-mooney you added the missing one
16:51:05 sean-k-mooney so i was more or less relying on thos but it hsould be easy to etend those to use the policy form the flavor
16:51:07 artom sean-k-mooney, well yeah, because we only have the alias based policies for now :)
16:52:03 sean-k-mooney ok ill push there version i have now after i adress you comments and ill start working on the func test after
16:54:04 openstackgerrit sean mooney proposed openstack/nova master: support pci numa affinity policies in flavor and image https://review.opendev.org/674072
17:09:10 openstackgerrit Eric Fried proposed openstack/nova master: Consolidate [image_cache] conf options https://review.opendev.org/690723
17:09:11 openstackgerrit Eric Fried proposed openstack/nova master: Add image caching to the support matrix https://review.opendev.org/690748
17:09:56 efried mriedem, stephenfin: Rebased and nit-fixed ^
17:12:15 openstackgerrit Stephen Finucane proposed openstack/nova master: "SUSPENDED" description changed in server_concepts guide and API REF https://review.opendev.org/663590
17:13:00 openstackgerrit Elod Illes proposed openstack/nova stable/pike: Error out interrupted builds https://review.opendev.org/687918
17:22:19 artom stephenfin, oh, friendly reminder that the NUMA LM func test stack is ready for another look: https://review.opendev.org/#/c/687404/
17:22:36 artom (I know it's not so much a learning curve as a learning brick wall)
17:24:10 stephenfin artom: Sure thing, but that's too much work for 5:20pm when jetlagged. I'll grab it in the morning :)
17:24:21 artom stephenfin, ack, thank you!
17:25:40 openstackgerrit Eric Fried proposed openstack/nova master: Remove functional test specific nova code https://review.opendev.org/683609
17:31:18 bauzas mriedem: dumb question but what the user sees from the API when we error out some instance because of NoValidHost ?
17:31:37 bauzas by a nova show I mean
17:32:00 bauzas mriedem: was about to say +1 to your ML thread, but was wondering the API result for the same
17:33:11 mriedem $ openstack server show build-fail1 -f value -c fault {u'message': u'No valid host was found. ', u'code': 500, u'created': u'2019-11-13T15:57:13Z'}
17:33:11 mriedem bauzas: the non-admin user? they see this:
17:33:27 mriedem the fault is only shown if the server status is ERROR or DELETED
17:33:46 bauzas mriedem: so the instance action event show should do the same
17:34:52 mriedem tbc, i wasn't proposing that the action event exception type is only shown for an ERROR or DELETED status server
17:35:08 mriedem since as i said in the thread, you can fail a resize and the server status doesn't go to ERROR
17:35:57 bauzas mriedem: yeah, I understood this
17:36:05 stephenfin efried: Think you could hold your nose on https://review.opendev.org/#/c/684345/16/nova/network/neutronv2/api.py to keep this moving, given I'm removing it again shortly after?
17:36:09 bauzas mriedem: that's why I think we should *also* do it
17:36:11 stephenfin drat, just missed him
17:36:37 bauzas and yeah, of course, when a user calls some ops, we can ask 'do a nova show'
17:36:58 bauzas but sometimes we need to look at the instance actions (like for resize) and that's why I'd love your spec :)
17:53:15 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'os-security-group-default-rules' REST API https://review.opendev.org/686807
17:53:15 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345
17:53:16 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove (most) '/os-networks' REST APIs https://review.opendev.org/686809
17:53:16 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused '*_default_rules' security group DB APIs https://review.opendev.org/686808
17:53:17 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests https://review.opendev.org/686811
17:53:17 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove '/os-tenant-networks' REST API https://review.opendev.org/686810
17:53:18 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove nova-manage network, floating commands https://review.opendev.org/686813
17:53:18 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812
17:53:19 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary https://review.opendev.org/686815
17:53:19 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs https://review.opendev.org/686814
17:53:20 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Blast most references to nova-network https://review.opendev.org/686817
17:53:20 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-network' binary https://review.opendev.org/686816
17:53:21 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP https://review.opendev.org/686818
20:08:05 openstackgerrit Dan Smith proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849
20:11:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Create instance action when burying in cell0 https://review.opendev.org/694165
20:27:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Create instance action when burying in cell0 https://review.opendev.org/694165
20:31:50 openstackgerrit Matt Riedemann proposed openstack/nova master: doc: add troubleshooting guide for cleaning up orphaned allocations https://review.opendev.org/691427
21:41:02 openstackgerrit François Palin proposed openstack/nova master: Add retry to cinder api calls related to volume detach https://review.opendev.org/669674
21:44:59 openstackgerrit Merged openstack/nova master: Rename Claims resources to compute_node https://review.opendev.org/679470
21:45:06 openstackgerrit Merged openstack/nova master: Clear instance.launched_on when build fails https://review.opendev.org/683725
21:49:12 melwitt so... after helping a colleague unwedge a failed resize for a customer, I've learned that we intentionally don't roll back port bindings to the source when finish_resize fails https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5274
21:49:51 melwitt and the reasoning is because the assumption is "everything is fine" on the dest and that only the virt guest needs to be created
21:50:54 melwitt but, when finish_resize fails because update of volume attachments fails (error had to do with "duplicate connectors") it doesn't seem like the dest should really be considered an ok place for the instance to be
21:57:39 melwitt I wonder if fixing this might be as simple as moving the volume attach update before the network setup
21:57:46 melwitt on the dest
21:58:29 mriedem i feel like i recently added that comment but maybe not
21:58:56 mriedem we have a lot of "cleanup networking and cleanup volumes" in live and cold migrate/resize where none of it is really atomic
21:59:36 mriedem i made a note of that in the revert code for cross-cell resize here https://review.opendev.org/#/c/637647/51/nova/compute/manager.py@4690
22:00:19 mriedem i'm not sure what that duplicate connectors error is
22:00:55 mriedem ah that comment is relatively new https://review.opendev.org/#/c/635349/
22:01:22 melwitt yeah we're going to be investigating the root cause of the duplicate connectors thing, never seen that before
22:01:49 melwitt error came back from cinder
22:01:59 mriedem what was the volume type?
22:02:51 melwitt I dunno yet. we only focused on recovery for now, it's late at night for those who were fixing the wedged resize
22:03:40 mriedem so migrate_instance_finish updated the port bindings to point at the dest host and then _update_volume_attachments blew up right?
22:03:43 melwitt we (they) moved the port bindings back to the source manually and then we hard rebooted the instance to get it going for the customer again. they had only wanted to resize to a bigger flavor
22:03:47 melwitt yes
22:04:03 mriedem still, if you get to finish_resize the instance.host already points at the dest
22:04:12 mriedem so they would have had to do more than change port bindings
22:04:17 mriedem to get the guest to reboot on the source
22:04:32 melwitt and it started up fine with the larger flavor once the port binding was put back on the source. and prior to that yes I had them change the host/node back to the source
22:04:32 mriedem resize_instance on the source updates the instance.host/node before casting to finish_resize on the dest
22:04:38 melwitt yeah sorry

Earlier   Later