Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-30
16:12:41 mnaser instances is shown when listing instances (with ACTIVE state), getting the specific instance (nova show/delete/etc) gives 404, instance_mapping exists, it exists in instances table in the appropriate cell
16:12:51 mnaser and no build request
16:12:53 mnaser so things.. should be right?
16:15:45 Guest9714 mnaser: instance_mapping with cell_id=None?
16:16:03 mnaser melwitt: nope, it has an assigned cell_id
16:16:09 mnaser and it exists in the assigned cell too
16:16:27 melwitt but a 404 during delete? hm
16:16:34 mnaser even 404 during get or anything
16:16:52 melwitt yeah, that's unexpected
16:17:06 melwitt (and wrong)
16:17:08 mnaser i restarted the nova apis... thought maybe it was cache but it wasnt
16:17:31 mnaser i mean afaik this should work .. https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2353-L2357
16:18:41 mriedem mnaser: if the instance mapping failed to update the record with the cell mapping we are going to fail to target the instance in any cell,
16:18:47 mriedem since we do'nt know which cell since the instance mapping is busted
16:19:06 mnaser mriedem: but i manaully updated the instance_mapping in my cleanup, so the cell_id is correct in instance_mapping
16:19:07 melwitt he said the instance mapping has a cell_id assigned
16:19:26 mriedem what's the stacktrace on the 404 then?
16:19:30 mriedem or i suppose there isn't one
16:19:38 mnaser none, just an instance not found
16:20:30 mnaser mriedem: http://paste.openstack.org/show/726855/
16:20:59 mnaser and cell_id=5 is the one that uses 'nova' db
16:22:20 mriedem and you're sure you don't have any services table entries with nova-osapi_compute that are <15/
16:22:21 mriedem ?
16:23:37 mriedem it would be looking in whatever [database]/connection database you're running for the nova.conf in the nova-api service
16:23:43 mriedem so presumably cell0
16:23:57 Sundar efried: Please ping me when you have the time. I am open for the next half hour.
16:24:10 efried Sundar: I'm here. Shall we talk in -cyborg?
16:24:13 mriedem note that min service version *is* cached
16:24:21 Sundar Sure
16:25:17 mnaser mriedem: http://paste.openstack.org/show/726857/
16:25:19 mnaser i'm pretty sure :X
16:25:27 mnaser there are deleted service records < 15
16:26:38 mnaser interesting
16:26:44 mriedem mnaser: what db is your api config using?
16:26:46 mriedem nova_cell0?
16:26:49 mnaser nova_api
16:26:53 mriedem no,
16:26:57 mriedem [database]/connection
16:27:01 mnaser nova
16:27:16 mnaser also i did a log.error(instance_uuid inst_map)
16:27:18 mnaser 729b5fb6-064c-4db4-b448-e01f5bc3d789: InstanceMapping(cell_mapping=CellMapping(e16503e7-b7dc-4f70-8c3e-87653c01481a),created_at=2018-07-30T15:31:56Z,id=8553936,instance_uuid=729b5fb6-064c-4db4-b448-e01f5bc3d789,project_id='55e74b674f714b869d14eb40ca004eb4',updated_at=2018-07-30T15:31:57Z)
16:27:20 mnaser so it actually gets it
16:28:11 mnaser adding a LOG.debug(instance) in the follow up
16:30:46 mnaser https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2355
16:30:48 mnaser it reaches here
16:31:02 mnaser but then `objects.Instance.get_by_uuid` raises the exception i think
16:31:06 mriedem and you should see get_or_set_cached_cell_and_set_connections in the logs for the lock
16:31:15 mriedem and it will log the cell mapping uuid
16:31:47 mriedem at debug i mean
16:31:53 mnaser well, I aded a LOG.error(inst_map.cell_mapping) and i see the cell mapping there on the request
16:32:11 mnaser so i guess objects.Instance.get_by_uuid is raising some exception
16:32:42 mnaser also
16:32:45 mriedem can you also log the context.db_connection right before Instance.get_by_uuid?
16:32:51 mnaser sure
16:32:52 mnaser https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2398-L2400
16:32:54 mnaser i think this is dead code
16:32:57 mriedem it won't be a string, but needs to be not None
16:33:09 melwitt usually, the only exception there would be InstanceNotFound but if deleted != 0 you wouldn't see it in 'nova list' either
16:33:17 mriedem mnaser: "# NOTE(ameade): we still need to support integer ids for ec2"
16:33:32 mnaser well the exception handles InvalidID
16:33:55 mnaser but if uuidutils.is_uuid_like is raises that then you never end up in else:
16:34:01 mnaser and if it doesn't, then InvalidID might never be raised
16:34:10 mnaser but idk, it just seems a bit weird but anyways
16:34:22 mnaser let me check context.db_connection
16:35:43 mnaser mriedem: <oslo_db.sqlalchemy.enginefacade._TransactionContextManager object at 0x955e450>
16:35:50 mriedem ok
16:35:56 mnaser i mean i could try to remember how to make sqlalchemy log sql queries
16:36:01 mnaser i remember i did that at some point
16:37:13 mriedem mnaser: it's either https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace i think
16:38:00 mriedem oh you know what...
16:38:04 mriedem i bet i know what this is
16:38:13 mriedem lazy-join on a deleted compute service by the instance record
16:38:18 mriedem it's not the instance record that's the problem
16:38:24 mriedem it's a related deleted service record that we can't load
16:38:31 mriedem melwitt: remember that^ ?
16:39:03 mriedem mnaser: i'd query the db that the instance is for all nova-compute services with the same name as the instance.host
16:39:07 mriedem are any of those deleted?
16:39:40 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use oslotest CaptureOutput fixture https://review.openstack.org/587129
16:39:41 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use a non-nova log capture fixture https://review.openstack.org/587130
16:39:42 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use a simplified WarningsFixture https://review.openstack.org/587131
16:39:44 mriedem it could be trying to set a uuid on a deleted service record
16:39:55 melwitt yeah... thinking. the one I've got a patch up for is around generating a uuid for a deleted service record
16:40:06 mriedem but that was with listing i thought, not sohw
16:40:08 mriedem *show
16:40:36 mriedem mnaser: i'd probably put a try/except around that Instance.get_by_uuid to handle InstanceNotFound, LOG.exception and re-raise
16:40:45 mriedem mnaser: to figure out where the InstanceNotFound originates from
16:41:59 melwitt yeah, according to the bugs, that would raise ServiceNotFound or ServiceTooOld for listing
16:46:49 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove superfluous network stubbing in func test https://review.openstack.org/587134
16:49:39 anupn Hi Folks, I have deployed an instance on ironic node, but that instance is unable to ping to the external network. Can you please suggest what things should I check for? my br-ex has got an external IP, and I can ping the internet from the host but not from the vm.
16:51:28 melwitt anupn: hi, this is a development channel, not a support channel. pls see topic
16:52:49 anupn melwitt: Ah ok! Will ask in the other channel
16:59:45 mnaser sorry, in a call, ill catch up on buffer in a sec
17:07:44 openstackgerrit melanie witt proposed openstack/nova master: Set default of oslo.privsep.daemon logging to INFO level https://review.openstack.org/586643
17:13:48 mnaser ill do the try/raise, the service is actually existing
17:22:17 mnaser http://paste.openstack.org/show/726864/
17:23:13 mnaser i feel like i need to take a step back and something weird is happening
17:23:37 openstackgerrit Chris Dent proposed openstack/nova master: DNM: Explore limiting resource tracker update https://review.openstack.org/587050
17:23:38 openstackgerrit Chris Dent proposed openstack/nova master: Extract _update_to_placement method in resource tracker https://review.openstack.org/587171
17:25:18 melwitt mnaser: so the deleted column for that instance != 0? I don't get how it's showing up in 'nova list' then
17:31:33 mnaser melwitt: also btw i did some work here on trying to make it a single transaction (with a parent fix trying to be a less major change) -- https://review.openstack.org/#/c/586742/ if you have time to look at it

Earlier   Later