| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-30 | |||
| 16:18:47 | mriedem | since we do'nt know which cell since the instance mapping is busted | |
| 16:19:06 | mnaser | mriedem: but i manaully updated the instance_mapping in my cleanup, so the cell_id is correct in instance_mapping | |
| 16:19:07 | melwitt | he said the instance mapping has a cell_id assigned | |
| 16:19:26 | mriedem | what's the stacktrace on the 404 then? | |
| 16:19:30 | mriedem | or i suppose there isn't one | |
| 16:19:38 | mnaser | none, just an instance not found | |
| 16:20:30 | mnaser | mriedem: http://paste.openstack.org/show/726855/ | |
| 16:20:59 | mnaser | and cell_id=5 is the one that uses 'nova' db | |
| 16:22:20 | mriedem | and you're sure you don't have any services table entries with nova-osapi_compute that are <15/ | |
| 16:22:21 | mriedem | ? | |
| 16:23:37 | mriedem | it would be looking in whatever [database]/connection database you're running for the nova.conf in the nova-api service | |
| 16:23:43 | mriedem | so presumably cell0 | |
| 16:23:57 | Sundar | efried: Please ping me when you have the time. I am open for the next half hour. | |
| 16:24:10 | efried | Sundar: I'm here. Shall we talk in -cyborg? | |
| 16:24:13 | mriedem | note that min service version *is* cached | |
| 16:24:21 | Sundar | Sure | |
| 16:25:17 | mnaser | mriedem: http://paste.openstack.org/show/726857/ | |
| 16:25:19 | mnaser | i'm pretty sure :X | |
| 16:25:27 | mnaser | there are deleted service records < 15 | |
| 16:26:38 | mnaser | interesting | |
| 16:26:44 | mriedem | mnaser: what db is your api config using? | |
| 16:26:46 | mriedem | nova_cell0? | |
| 16:26:49 | mnaser | nova_api | |
| 16:26:53 | mriedem | no, | |
| 16:26:57 | mriedem | [database]/connection | |
| 16:27:01 | mnaser | nova | |
| 16:27:16 | mnaser | also i did a log.error(instance_uuid inst_map) | |
| 16:27:18 | mnaser | 729b5fb6-064c-4db4-b448-e01f5bc3d789: InstanceMapping(cell_mapping=CellMapping(e16503e7-b7dc-4f70-8c3e-87653c01481a),created_at=2018-07-30T15:31:56Z,id=8553936,instance_uuid=729b5fb6-064c-4db4-b448-e01f5bc3d789,project_id='55e74b674f714b869d14eb40ca004eb4',updated_at=2018-07-30T15:31:57Z) | |
| 16:27:20 | mnaser | so it actually gets it | |
| 16:28:11 | mnaser | adding a LOG.debug(instance) in the follow up | |
| 16:30:46 | mnaser | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2355 | |
| 16:30:48 | mnaser | it reaches here | |
| 16:31:02 | mnaser | but then `objects.Instance.get_by_uuid` raises the exception i think | |
| 16:31:06 | mriedem | and you should see get_or_set_cached_cell_and_set_connections in the logs for the lock | |
| 16:31:15 | mriedem | and it will log the cell mapping uuid | |
| 16:31:47 | mriedem | at debug i mean | |
| 16:31:53 | mnaser | well, I aded a LOG.error(inst_map.cell_mapping) and i see the cell mapping there on the request | |
| 16:32:11 | mnaser | so i guess objects.Instance.get_by_uuid is raising some exception | |
| 16:32:42 | mnaser | also | |
| 16:32:45 | mriedem | can you also log the context.db_connection right before Instance.get_by_uuid? | |
| 16:32:51 | mnaser | sure | |
| 16:32:52 | mnaser | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2398-L2400 | |
| 16:32:54 | mnaser | i think this is dead code | |
| 16:32:57 | mriedem | it won't be a string, but needs to be not None | |
| 16:33:09 | melwitt | usually, the only exception there would be InstanceNotFound but if deleted != 0 you wouldn't see it in 'nova list' either | |
| 16:33:17 | mriedem | mnaser: "# NOTE(ameade): we still need to support integer ids for ec2" | |
| 16:33:32 | mnaser | well the exception handles InvalidID | |
| 16:33:55 | mnaser | but if uuidutils.is_uuid_like is raises that then you never end up in else: | |
| 16:34:01 | mnaser | and if it doesn't, then InvalidID might never be raised | |
| 16:34:10 | mnaser | but idk, it just seems a bit weird but anyways | |
| 16:34:22 | mnaser | let me check context.db_connection | |
| 16:35:43 | mnaser | mriedem: <oslo_db.sqlalchemy.enginefacade._TransactionContextManager object at 0x955e450> | |
| 16:35:50 | mriedem | ok | |
| 16:35:56 | mnaser | i mean i could try to remember how to make sqlalchemy log sql queries | |
| 16:36:01 | mnaser | i remember i did that at some point | |
| 16:37:13 | mriedem | mnaser: it's either https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace i think | |
| 16:38:00 | mriedem | oh you know what... | |
| 16:38:04 | mriedem | i bet i know what this is | |
| 16:38:13 | mriedem | lazy-join on a deleted compute service by the instance record | |
| 16:38:18 | mriedem | it's not the instance record that's the problem | |
| 16:38:24 | mriedem | it's a related deleted service record that we can't load | |
| 16:38:31 | mriedem | melwitt: remember that^ ? | |
| 16:39:03 | mriedem | mnaser: i'd query the db that the instance is for all nova-compute services with the same name as the instance.host | |
| 16:39:07 | mriedem | are any of those deleted? | |
| 16:39:40 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Use oslotest CaptureOutput fixture https://review.openstack.org/587129 | |
| 16:39:41 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Use a non-nova log capture fixture https://review.openstack.org/587130 | |
| 16:39:42 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Use a simplified WarningsFixture https://review.openstack.org/587131 | |
| 16:39:44 | mriedem | it could be trying to set a uuid on a deleted service record | |
| 16:39:55 | melwitt | yeah... thinking. the one I've got a patch up for is around generating a uuid for a deleted service record | |
| 16:40:06 | mriedem | but that was with listing i thought, not sohw | |
| 16:40:08 | mriedem | *show | |
| 16:40:36 | mriedem | mnaser: i'd probably put a try/except around that Instance.get_by_uuid to handle InstanceNotFound, LOG.exception and re-raise | |
| 16:40:45 | mriedem | mnaser: to figure out where the InstanceNotFound originates from | |
| 16:41:59 | melwitt | yeah, according to the bugs, that would raise ServiceNotFound or ServiceTooOld for listing | |
| 16:46:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove superfluous network stubbing in func test https://review.openstack.org/587134 | |
| 16:49:39 | anupn | Hi Folks, I have deployed an instance on ironic node, but that instance is unable to ping to the external network. Can you please suggest what things should I check for? my br-ex has got an external IP, and I can ping the internet from the host but not from the vm. | |
| 16:51:28 | melwitt | anupn: hi, this is a development channel, not a support channel. pls see topic | |
| 16:52:49 | anupn | melwitt: Ah ok! Will ask in the other channel | |
| 16:59:45 | mnaser | sorry, in a call, ill catch up on buffer in a sec | |
| 17:07:44 | openstackgerrit | melanie witt proposed openstack/nova master: Set default of oslo.privsep.daemon logging to INFO level https://review.openstack.org/586643 | |
| 17:13:48 | mnaser | ill do the try/raise, the service is actually existing | |
| 17:22:17 | mnaser | http://paste.openstack.org/show/726864/ | |
| 17:23:13 | mnaser | i feel like i need to take a step back and something weird is happening | |
| 17:23:37 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Explore limiting resource tracker update https://review.openstack.org/587050 | |
| 17:23:38 | openstackgerrit | Chris Dent proposed openstack/nova master: Extract _update_to_placement method in resource tracker https://review.openstack.org/587171 | |
| 17:25:18 | melwitt | mnaser: so the deleted column for that instance != 0? I don't get how it's showing up in 'nova list' then | |
| 17:31:33 | mnaser | melwitt: also btw i did some work here on trying to make it a single transaction (with a parent fix trying to be a less major change) -- https://review.openstack.org/#/c/586742/ if you have time to look at it | |
| 17:31:56 | melwitt | mnaser: will take a look, thanks | |
| 17:39:41 | openstackgerrit | Merged openstack/nova stable/pike: libvirt: add method to configure migration speed https://review.openstack.org/586965 | |
| 17:54:42 | openstackgerrit | Merged openstack/nova stable/pike: Make host_aggregate_map dictionary case-insensitive https://review.openstack.org/585335 | |
| 17:54:50 | openstackgerrit | Merged openstack/nova stable/pike: Fix unbound local when saving an unchanged RequestSpec https://review.openstack.org/580951 | |
| 17:54:56 | openstackgerrit | Merged openstack/nova stable/queens: Refactor _build_device_metadata https://review.openstack.org/581398 | |
| 17:55:05 | openstackgerrit | Merged openstack/nova stable/queens: Consider hostdev devices when building metadata https://review.openstack.org/581399 | |
| 17:55:13 | openstackgerrit | Merged openstack/nova master: Add additional functional tests for NUMA networks https://review.openstack.org/585385 | |
| 17:59:20 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Use of set_middleware_defaults https://review.openstack.org/587183 | |
| 18:00:32 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Use own set_middleware_defaults https://review.openstack.org/587183 | |
| 18:32:33 | efried | Hey folks, we're back with https://bugs.launchpad.net/nova/+bug/1737711 | |
| 18:32:33 | openstack | Launchpad bug 1737711 in OpenStack Compute (nova) "nova boot failed when use the chinese metadata key and value" [Undecided,In progress] - Assigned to huanhongda (hongda) | |
| 18:32:59 | efried | They've restricted the patch https://review.openstack.org/#/c/536236/ to just metadata keys (not extra_specs keys anymore). | |
| 18:34:20 | efried | The bug still doesn't give a whole lot of detail as to the use case for this, but maybe it makes more sense to y'all, dansmith mriedem jaypipes | |