Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-30
16:19:26 mriedem what's the stacktrace on the 404 then?
16:19:30 mriedem or i suppose there isn't one
16:19:38 mnaser none, just an instance not found
16:20:30 mnaser mriedem: http://paste.openstack.org/show/726855/
16:20:59 mnaser and cell_id=5 is the one that uses 'nova' db
16:22:20 mriedem and you're sure you don't have any services table entries with nova-osapi_compute that are <15/
16:22:21 mriedem ?
16:23:37 mriedem it would be looking in whatever [database]/connection database you're running for the nova.conf in the nova-api service
16:23:43 mriedem so presumably cell0
16:23:57 Sundar efried: Please ping me when you have the time. I am open for the next half hour.
16:24:10 efried Sundar: I'm here. Shall we talk in -cyborg?
16:24:13 mriedem note that min service version *is* cached
16:24:21 Sundar Sure
16:25:17 mnaser mriedem: http://paste.openstack.org/show/726857/
16:25:19 mnaser i'm pretty sure :X
16:25:27 mnaser there are deleted service records < 15
16:26:38 mnaser interesting
16:26:44 mriedem mnaser: what db is your api config using?
16:26:46 mriedem nova_cell0?
16:26:49 mnaser nova_api
16:26:53 mriedem no,
16:26:57 mriedem [database]/connection
16:27:01 mnaser nova
16:27:16 mnaser also i did a log.error(instance_uuid inst_map)
16:27:18 mnaser 729b5fb6-064c-4db4-b448-e01f5bc3d789: InstanceMapping(cell_mapping=CellMapping(e16503e7-b7dc-4f70-8c3e-87653c01481a),created_at=2018-07-30T15:31:56Z,id=8553936,instance_uuid=729b5fb6-064c-4db4-b448-e01f5bc3d789,project_id='55e74b674f714b869d14eb40ca004eb4',updated_at=2018-07-30T15:31:57Z)
16:27:20 mnaser so it actually gets it
16:28:11 mnaser adding a LOG.debug(instance) in the follow up
16:30:46 mnaser https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2355
16:30:48 mnaser it reaches here
16:31:02 mnaser but then `objects.Instance.get_by_uuid` raises the exception i think
16:31:06 mriedem and you should see get_or_set_cached_cell_and_set_connections in the logs for the lock
16:31:15 mriedem and it will log the cell mapping uuid
16:31:47 mriedem at debug i mean
16:31:53 mnaser well, I aded a LOG.error(inst_map.cell_mapping) and i see the cell mapping there on the request
16:32:11 mnaser so i guess objects.Instance.get_by_uuid is raising some exception
16:32:42 mnaser also
16:32:45 mriedem can you also log the context.db_connection right before Instance.get_by_uuid?
16:32:51 mnaser sure
16:32:52 mnaser https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2398-L2400
16:32:54 mnaser i think this is dead code
16:32:57 mriedem it won't be a string, but needs to be not None
16:33:09 melwitt usually, the only exception there would be InstanceNotFound but if deleted != 0 you wouldn't see it in 'nova list' either
16:33:17 mriedem mnaser: "# NOTE(ameade): we still need to support integer ids for ec2"
16:33:32 mnaser well the exception handles InvalidID
16:33:55 mnaser but if uuidutils.is_uuid_like is raises that then you never end up in else:
16:34:01 mnaser and if it doesn't, then InvalidID might never be raised
16:34:10 mnaser but idk, it just seems a bit weird but anyways
16:34:22 mnaser let me check context.db_connection
16:35:43 mnaser mriedem: <oslo_db.sqlalchemy.enginefacade._TransactionContextManager object at 0x955e450>
16:35:50 mriedem ok
16:35:56 mnaser i mean i could try to remember how to make sqlalchemy log sql queries
16:36:01 mnaser i remember i did that at some point
16:37:13 mriedem mnaser: it's either https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace i think
16:38:00 mriedem oh you know what...
16:38:04 mriedem i bet i know what this is
16:38:13 mriedem lazy-join on a deleted compute service by the instance record
16:38:18 mriedem it's not the instance record that's the problem
16:38:24 mriedem it's a related deleted service record that we can't load
16:38:31 mriedem melwitt: remember that^ ?
16:39:03 mriedem mnaser: i'd query the db that the instance is for all nova-compute services with the same name as the instance.host
16:39:07 mriedem are any of those deleted?
16:39:40 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use oslotest CaptureOutput fixture https://review.openstack.org/587129
16:39:41 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use a non-nova log capture fixture https://review.openstack.org/587130
16:39:42 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use a simplified WarningsFixture https://review.openstack.org/587131
16:39:44 mriedem it could be trying to set a uuid on a deleted service record
16:39:55 melwitt yeah... thinking. the one I've got a patch up for is around generating a uuid for a deleted service record
16:40:06 mriedem but that was with listing i thought, not sohw
16:40:08 mriedem *show
16:40:36 mriedem mnaser: i'd probably put a try/except around that Instance.get_by_uuid to handle InstanceNotFound, LOG.exception and re-raise
16:40:45 mriedem mnaser: to figure out where the InstanceNotFound originates from
16:41:59 melwitt yeah, according to the bugs, that would raise ServiceNotFound or ServiceTooOld for listing
16:46:49 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove superfluous network stubbing in func test https://review.openstack.org/587134
16:49:39 anupn Hi Folks, I have deployed an instance on ironic node, but that instance is unable to ping to the external network. Can you please suggest what things should I check for? my br-ex has got an external IP, and I can ping the internet from the host but not from the vm.
16:51:28 melwitt anupn: hi, this is a development channel, not a support channel. pls see topic
16:52:49 anupn melwitt: Ah ok! Will ask in the other channel
16:59:45 mnaser sorry, in a call, ill catch up on buffer in a sec
17:07:44 openstackgerrit melanie witt proposed openstack/nova master: Set default of oslo.privsep.daemon logging to INFO level https://review.openstack.org/586643
17:13:48 mnaser ill do the try/raise, the service is actually existing
17:22:17 mnaser http://paste.openstack.org/show/726864/
17:23:13 mnaser i feel like i need to take a step back and something weird is happening
17:23:37 openstackgerrit Chris Dent proposed openstack/nova master: DNM: Explore limiting resource tracker update https://review.openstack.org/587050
17:23:38 openstackgerrit Chris Dent proposed openstack/nova master: Extract _update_to_placement method in resource tracker https://review.openstack.org/587171
17:25:18 melwitt mnaser: so the deleted column for that instance != 0? I don't get how it's showing up in 'nova list' then
17:31:33 mnaser melwitt: also btw i did some work here on trying to make it a single transaction (with a parent fix trying to be a less major change) -- https://review.openstack.org/#/c/586742/ if you have time to look at it
17:31:56 melwitt mnaser: will take a look, thanks
17:39:41 openstackgerrit Merged openstack/nova stable/pike: libvirt: add method to configure migration speed https://review.openstack.org/586965
17:54:42 openstackgerrit Merged openstack/nova stable/pike: Make host_aggregate_map dictionary case-insensitive https://review.openstack.org/585335
17:54:50 openstackgerrit Merged openstack/nova stable/pike: Fix unbound local when saving an unchanged RequestSpec https://review.openstack.org/580951
17:54:56 openstackgerrit Merged openstack/nova stable/queens: Refactor _build_device_metadata https://review.openstack.org/581398
17:55:05 openstackgerrit Merged openstack/nova stable/queens: Consider hostdev devices when building metadata https://review.openstack.org/581399
17:55:13 openstackgerrit Merged openstack/nova master: Add additional functional tests for NUMA networks https://review.openstack.org/585385
17:59:20 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use of set_middleware_defaults https://review.openstack.org/587183
18:00:32 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use own set_middleware_defaults https://review.openstack.org/587183
18:32:33 openstack Launchpad bug 1737711 in OpenStack Compute (nova) "nova boot failed when use the chinese metadata key and value" [Undecided,In progress] - Assigned to huanhongda (hongda)
18:32:33 efried Hey folks, we're back with https://bugs.launchpad.net/nova/+bug/1737711
18:32:59 efried They've restricted the patch https://review.openstack.org/#/c/536236/ to just metadata keys (not extra_specs keys anymore).
18:34:20 efried The bug still doesn't give a whole lot of detail as to the use case for this, but maybe it makes more sense to y'all, dansmith mriedem jaypipes
18:38:21 melwitt efried: fwiw, my initial read on that is we have more than one place where unicode keys won't work (as you mentioned extra_specs is another one) and IMHO it feels like it would be a small spec to propose to change all of them together in a consistent way to support unicode
18:38:39 melwitt is the schema validation really the only thing preventing unicode from working?
18:39:14 efried melwitt: Good question. No idea if they've tried... whatever it is they wanted to try, with their fix in place.

Earlier   Later