| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-30 | |||
| 14:39:12 | dansmith | alright | |
| 14:39:51 | kashyap | mnaser: Maybe it's just me, but first paragraph in the commit message is confusing | |
| 14:40:52 | kashyap | (And the patch above needs to be rebased I think) | |
| 14:40:56 | dansmith | makes sense to me | |
| 14:42:06 | kashyap | You have more context, though :-) But yeah, reading the whole message makes it clearer. | |
| 14:44:58 | mnaser | kashyap: yeah, it's still very much early and needs reviews :) | |
| 14:45:08 | mnaser | it was more of a rough "hey how's this to solve a problem" | |
| 14:46:37 | dansmith | gibi: the fix for the other test was in the create_port() handler of the fixture, so it was touched by this test and part of your other feedback anyway, | |
| 14:46:47 | dansmith | gibi: so I'll just squash and update this patch | |
| 14:46:53 | gibi | dansmith: OK, cool | |
| 14:47:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Add additional functional tests for NUMA networks https://review.openstack.org/585385 | |
| 14:47:13 | dansmith | gibi: note that just moves it up one test level, not all the way to the top | |
| 14:48:14 | kashyap | mdbooth: Not sure if you'll be able to today, but this is ready and fixed all the tests (including the live migration job): https://review.openstack.org/#/c/567258/ | |
| 14:48:33 | mdbooth | kashyap: ack | |
| 14:48:37 | kashyap | Gentle ping because you've reviewed it before. (And I'm off for 2 weeks from Wed.) | |
| 14:48:41 | kashyap | mdbooth: Thanks! | |
| 14:50:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Updated AggregateImagePropertiesIsolation filter illustration https://review.openstack.org/586317 | |
| 14:51:55 | tetsuro_ | gibi: I'm going to rebase https://review.openstack.org/#/c/585672/ on gibi's 1.28 support one. | |
| 14:52:00 | openstack | Launchpad bug 1783635 in OpenStack Compute (nova) "Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'" [Undecided,New] | |
| 14:52:00 | mriedem | lyarwood: looks like a backport to pike is using a method that doesn't even exist :( https://bugs.launchpad.net/nova/+bug/1783635 | |
| 14:52:09 | tetsuro_ | gibi: Before pushing them, I'm looking into the functional test in https://review.openstack.org/#/c/527728/18 to pass rebased on the two. | |
| 14:52:27 | tetsuro_ | gibi, lajos: Would you mind if I amend https://review.openstack.org/#/c/527728/18? | |
| 14:54:37 | mriedem | lyarwood: i guess you figured that out already https://review.openstack.org/#/c/586965/ | |
| 14:54:39 | gibi | tetsuro_: thanks for the heads-up. I have nothing agains the rebase. I talked to lajos before and he let me take over https://review.openstack.org/#/c/527728 so I think I can let you in his name to amend it :) | |
| 14:56:08 | tetsuro_ | gibi: Thanks! | |
| 14:56:11 | gibi | tetsuro_: regarding the rebase the patch adding support for 1.29 top of 1.28. That is also cool with me | |
| 14:56:45 | tetsuro_ | Yup. That would be the right way to go. | |
| 14:56:49 | gibi | tetsuro_: I will try to keep continue progressing support for 1.28 as my time allows during the RC period so we can merge it soon in Stein | |
| 14:58:51 | tetsuro_ | gibi: Cool. I can help the "change unit tests to pass" part if I finish the rebase and test work earlier than I think. | |
| 15:00:00 | gibi | tetsuro_: I consider unit test secondary in 1.28 but thanks for any help with those. I'm more affraid of how to add proper functional coverage for the consumer generation conflict cases | |
| 15:00:54 | tetsuro_ | gibi: yup, that sounds tough... | |
| 15:01:12 | mriedem | dansmith: https://review.openstack.org/#/c/586965/ to unbreak live migration in pike | |
| 15:01:36 | gibi | tetsuro_: I have some ideas but until I see them working I'm in doubt | |
| 15:01:51 | mriedem | well, live migration + pike + linuxbridge, which we don't test in CI | |
| 15:02:11 | dansmith | mriedem: is this because of that speed limit thing? | |
| 15:02:22 | openstackgerrit | Merged openstack/nova master: tox: Reuse envdirs https://review.openstack.org/534382 | |
| 15:02:22 | dansmith | for LB yeah | |
| 15:03:19 | tetsuro_ | gibi: But as many project (neutron, cyborg, blazar etc) are looking forward the feature we can just continue to work. | |
| 15:03:54 | tetsuro_ | gibi: Anyway, I should go to bed now, sorry. Thanks for your info. | |
| 15:04:17 | gibi | tetsuro_: yeah, not to mention that my work on bandwidth base scheduling also needs it :) | |
| 15:04:33 | gibi | tetsuro_: thanks for the info too. good night! | |
| 15:04:56 | tetsuro_ | gibi: Good day and night! | |
| 15:10:31 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add unshelve instance error info to fault table https://review.openstack.org/583426 | |
| 15:16:56 | gibi | dansmith: thanks for the fix in https://review.openstack.org/#/c/585385 I'm +2 now | |
| 15:17:05 | gibi | mriedem: ^^ you were +2 before | |
| 15:17:26 | dansmith | gibi: thanks | |
| 15:35:05 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Add unshelve instance error info to fault table https://review.openstack.org/583422 | |
| 16:11:59 | mnaser | well i sure don't run out of ways to find how broken this was :( | |
| 16:12:41 | mnaser | instances is shown when listing instances (with ACTIVE state), getting the specific instance (nova show/delete/etc) gives 404, instance_mapping exists, it exists in instances table in the appropriate cell | |
| 16:12:51 | mnaser | and no build request | |
| 16:12:53 | mnaser | so things.. should be right? | |
| 16:15:45 | Guest9714 | mnaser: instance_mapping with cell_id=None? | |
| 16:16:03 | mnaser | melwitt: nope, it has an assigned cell_id | |
| 16:16:09 | mnaser | and it exists in the assigned cell too | |
| 16:16:27 | melwitt | but a 404 during delete? hm | |
| 16:16:34 | mnaser | even 404 during get or anything | |
| 16:16:52 | melwitt | yeah, that's unexpected | |
| 16:17:06 | melwitt | (and wrong) | |
| 16:17:08 | mnaser | i restarted the nova apis... thought maybe it was cache but it wasnt | |
| 16:17:31 | mnaser | i mean afaik this should work .. https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2353-L2357 | |
| 16:18:41 | mriedem | mnaser: if the instance mapping failed to update the record with the cell mapping we are going to fail to target the instance in any cell, | |
| 16:18:47 | mriedem | since we do'nt know which cell since the instance mapping is busted | |
| 16:19:06 | mnaser | mriedem: but i manaully updated the instance_mapping in my cleanup, so the cell_id is correct in instance_mapping | |
| 16:19:07 | melwitt | he said the instance mapping has a cell_id assigned | |
| 16:19:26 | mriedem | what's the stacktrace on the 404 then? | |
| 16:19:30 | mriedem | or i suppose there isn't one | |
| 16:19:38 | mnaser | none, just an instance not found | |
| 16:20:30 | mnaser | mriedem: http://paste.openstack.org/show/726855/ | |
| 16:20:59 | mnaser | and cell_id=5 is the one that uses 'nova' db | |
| 16:22:20 | mriedem | and you're sure you don't have any services table entries with nova-osapi_compute that are <15/ | |
| 16:22:21 | mriedem | ? | |
| 16:23:37 | mriedem | it would be looking in whatever [database]/connection database you're running for the nova.conf in the nova-api service | |
| 16:23:43 | mriedem | so presumably cell0 | |
| 16:23:57 | Sundar | efried: Please ping me when you have the time. I am open for the next half hour. | |
| 16:24:10 | efried | Sundar: I'm here. Shall we talk in -cyborg? | |
| 16:24:13 | mriedem | note that min service version *is* cached | |
| 16:24:21 | Sundar | Sure | |
| 16:25:17 | mnaser | mriedem: http://paste.openstack.org/show/726857/ | |
| 16:25:19 | mnaser | i'm pretty sure :X | |
| 16:25:27 | mnaser | there are deleted service records < 15 | |
| 16:26:38 | mnaser | interesting | |
| 16:26:44 | mriedem | mnaser: what db is your api config using? | |
| 16:26:46 | mriedem | nova_cell0? | |
| 16:26:49 | mnaser | nova_api | |
| 16:26:53 | mriedem | no, | |
| 16:26:57 | mriedem | [database]/connection | |
| 16:27:01 | mnaser | nova | |
| 16:27:16 | mnaser | also i did a log.error(instance_uuid inst_map) | |
| 16:27:18 | mnaser | 729b5fb6-064c-4db4-b448-e01f5bc3d789: InstanceMapping(cell_mapping=CellMapping(e16503e7-b7dc-4f70-8c3e-87653c01481a),created_at=2018-07-30T15:31:56Z,id=8553936,instance_uuid=729b5fb6-064c-4db4-b448-e01f5bc3d789,project_id='55e74b674f714b869d14eb40ca004eb4',updated_at=2018-07-30T15:31:57Z) | |
| 16:27:20 | mnaser | so it actually gets it | |
| 16:28:11 | mnaser | adding a LOG.debug(instance) in the follow up | |
| 16:30:46 | mnaser | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2355 | |
| 16:30:48 | mnaser | it reaches here | |
| 16:31:02 | mnaser | but then `objects.Instance.get_by_uuid` raises the exception i think | |
| 16:31:06 | mriedem | and you should see get_or_set_cached_cell_and_set_connections in the logs for the lock | |
| 16:31:15 | mriedem | and it will log the cell mapping uuid | |
| 16:31:47 | mriedem | at debug i mean | |
| 16:31:53 | mnaser | well, I aded a LOG.error(inst_map.cell_mapping) and i see the cell mapping there on the request | |
| 16:32:11 | mnaser | so i guess objects.Instance.get_by_uuid is raising some exception | |
| 16:32:42 | mnaser | also | |