Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-30
14:52:00 mriedem lyarwood: looks like a backport to pike is using a method that doesn't even exist :( https://bugs.launchpad.net/nova/+bug/1783635
14:52:09 tetsuro_ gibi: Before pushing them, I'm looking into the functional test in https://review.openstack.org/#/c/527728/18 to pass rebased on the two.
14:52:27 tetsuro_ gibi, lajos: Would you mind if I amend https://review.openstack.org/#/c/527728/18?
14:54:37 mriedem lyarwood: i guess you figured that out already https://review.openstack.org/#/c/586965/
14:54:39 gibi tetsuro_: thanks for the heads-up. I have nothing agains the rebase. I talked to lajos before and he let me take over https://review.openstack.org/#/c/527728 so I think I can let you in his name to amend it :)
14:56:08 tetsuro_ gibi: Thanks!
14:56:11 gibi tetsuro_: regarding the rebase the patch adding support for 1.29 top of 1.28. That is also cool with me
14:56:45 tetsuro_ Yup. That would be the right way to go.
14:56:49 gibi tetsuro_: I will try to keep continue progressing support for 1.28 as my time allows during the RC period so we can merge it soon in Stein
14:58:51 tetsuro_ gibi: Cool. I can help the "change unit tests to pass" part if I finish the rebase and test work earlier than I think.
15:00:00 gibi tetsuro_: I consider unit test secondary in 1.28 but thanks for any help with those. I'm more affraid of how to add proper functional coverage for the consumer generation conflict cases
15:00:54 tetsuro_ gibi: yup, that sounds tough...
15:01:12 mriedem dansmith: https://review.openstack.org/#/c/586965/ to unbreak live migration in pike
15:01:36 gibi tetsuro_: I have some ideas but until I see them working I'm in doubt
15:01:51 mriedem well, live migration + pike + linuxbridge, which we don't test in CI
15:02:11 dansmith mriedem: is this because of that speed limit thing?
15:02:22 openstackgerrit Merged openstack/nova master: tox: Reuse envdirs https://review.openstack.org/534382
15:02:22 dansmith for LB yeah
15:03:19 tetsuro_ gibi: But as many project (neutron, cyborg, blazar etc) are looking forward the feature we can just continue to work.
15:03:54 tetsuro_ gibi: Anyway, I should go to bed now, sorry. Thanks for your info.
15:04:17 gibi tetsuro_: yeah, not to mention that my work on bandwidth base scheduling also needs it :)
15:04:33 gibi tetsuro_: thanks for the info too. good night!
15:04:56 tetsuro_ gibi: Good day and night!
15:10:31 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add unshelve instance error info to fault table https://review.openstack.org/583426
15:16:56 gibi dansmith: thanks for the fix in https://review.openstack.org/#/c/585385 I'm +2 now
15:17:05 gibi mriedem: ^^ you were +2 before
15:17:26 dansmith gibi: thanks
15:35:05 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Add unshelve instance error info to fault table https://review.openstack.org/583422
16:11:59 mnaser well i sure don't run out of ways to find how broken this was :(
16:12:41 mnaser instances is shown when listing instances (with ACTIVE state), getting the specific instance (nova show/delete/etc) gives 404, instance_mapping exists, it exists in instances table in the appropriate cell
16:12:51 mnaser and no build request
16:12:53 mnaser so things.. should be right?
16:15:45 Guest9714 mnaser: instance_mapping with cell_id=None?
16:16:03 mnaser melwitt: nope, it has an assigned cell_id
16:16:09 mnaser and it exists in the assigned cell too
16:16:27 melwitt but a 404 during delete? hm
16:16:34 mnaser even 404 during get or anything
16:16:52 melwitt yeah, that's unexpected
16:17:06 melwitt (and wrong)
16:17:08 mnaser i restarted the nova apis... thought maybe it was cache but it wasnt
16:17:31 mnaser i mean afaik this should work .. https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2353-L2357
16:18:41 mriedem mnaser: if the instance mapping failed to update the record with the cell mapping we are going to fail to target the instance in any cell,
16:18:47 mriedem since we do'nt know which cell since the instance mapping is busted
16:19:06 mnaser mriedem: but i manaully updated the instance_mapping in my cleanup, so the cell_id is correct in instance_mapping
16:19:07 melwitt he said the instance mapping has a cell_id assigned
16:19:26 mriedem what's the stacktrace on the 404 then?
16:19:30 mriedem or i suppose there isn't one
16:19:38 mnaser none, just an instance not found
16:20:30 mnaser mriedem: http://paste.openstack.org/show/726855/
16:20:59 mnaser and cell_id=5 is the one that uses 'nova' db
16:22:20 mriedem and you're sure you don't have any services table entries with nova-osapi_compute that are <15/
16:22:21 mriedem ?
16:23:37 mriedem it would be looking in whatever [database]/connection database you're running for the nova.conf in the nova-api service
16:23:43 mriedem so presumably cell0
16:23:57 Sundar efried: Please ping me when you have the time. I am open for the next half hour.
16:24:10 efried Sundar: I'm here. Shall we talk in -cyborg?
16:24:13 mriedem note that min service version *is* cached
16:24:21 Sundar Sure
16:25:17 mnaser mriedem: http://paste.openstack.org/show/726857/
16:25:19 mnaser i'm pretty sure :X
16:25:27 mnaser there are deleted service records < 15
16:26:38 mnaser interesting
16:26:44 mriedem mnaser: what db is your api config using?
16:26:46 mriedem nova_cell0?
16:26:49 mnaser nova_api
16:26:53 mriedem no,
16:26:57 mriedem [database]/connection
16:27:01 mnaser nova
16:27:16 mnaser also i did a log.error(instance_uuid inst_map)
16:27:18 mnaser 729b5fb6-064c-4db4-b448-e01f5bc3d789: InstanceMapping(cell_mapping=CellMapping(e16503e7-b7dc-4f70-8c3e-87653c01481a),created_at=2018-07-30T15:31:56Z,id=8553936,instance_uuid=729b5fb6-064c-4db4-b448-e01f5bc3d789,project_id='55e74b674f714b869d14eb40ca004eb4',updated_at=2018-07-30T15:31:57Z)
16:27:20 mnaser so it actually gets it
16:28:11 mnaser adding a LOG.debug(instance) in the follow up
16:30:46 mnaser https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2355
16:30:48 mnaser it reaches here
16:31:02 mnaser but then `objects.Instance.get_by_uuid` raises the exception i think
16:31:06 mriedem and you should see get_or_set_cached_cell_and_set_connections in the logs for the lock
16:31:15 mriedem and it will log the cell mapping uuid
16:31:47 mriedem at debug i mean
16:31:53 mnaser well, I aded a LOG.error(inst_map.cell_mapping) and i see the cell mapping there on the request
16:32:11 mnaser so i guess objects.Instance.get_by_uuid is raising some exception
16:32:42 mnaser also
16:32:45 mriedem can you also log the context.db_connection right before Instance.get_by_uuid?
16:32:51 mnaser sure
16:32:52 mnaser https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2398-L2400
16:32:54 mnaser i think this is dead code
16:32:57 mriedem it won't be a string, but needs to be not None
16:33:09 melwitt usually, the only exception there would be InstanceNotFound but if deleted != 0 you wouldn't see it in 'nova list' either
16:33:17 mriedem mnaser: "# NOTE(ameade): we still need to support integer ids for ec2"
16:33:32 mnaser well the exception handles InvalidID
16:33:55 mnaser but if uuidutils.is_uuid_like is raises that then you never end up in else:
16:34:01 mnaser and if it doesn't, then InvalidID might never be raised
16:34:10 mnaser but idk, it just seems a bit weird but anyways
16:34:22 mnaser let me check context.db_connection
16:35:43 mnaser mriedem: <oslo_db.sqlalchemy.enginefacade._TransactionContextManager object at 0x955e450>
16:35:50 mriedem ok
16:35:56 mnaser i mean i could try to remember how to make sqlalchemy log sql queries
16:36:01 mnaser i remember i did that at some point
16:37:13 mriedem mnaser: it's either https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace i think
16:38:00 mriedem oh you know what...
16:38:04 mriedem i bet i know what this is

Earlier   Later