| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-06 | |||
| 15:40:51 | efried | But in the meantime, <shrug> let's see if using the rootwrap daemon helps. | |
| 15:44:06 | stephenfin | jaypipes: https://review.openstack.org/#/c/498817/ and friends have their gate +1 now, if you were idle enough :) | |
| 15:56:19 | dansmith | mriedem: yeah I had marked that as -W ahead of me writing the release note, but then I wiped it when I pushed a little fix | |
| 15:57:00 | mriedem | i don't know how the tests are even passing | |
| 15:57:12 | dansmith | mriedem: because you missed that there is a patch before this? | |
| 15:57:41 | mriedem | yes | |
| 16:00:14 | esberglu | efried: ^ 459166 just got W+1 | |
| 16:00:59 | esberglu | efried: Oh wait nvm looking at the wrong thing | |
| 16:01:25 | efried | esberglu Ya know, we could patch that guy in and make our own patch that uses it... | |
| 16:05:33 | openstackgerrit | Merged openstack/nova master: Remove unneeded attributes from context https://review.openstack.org/498615 | |
| 16:06:14 | openstackgerrit | Merged openstack/nova master: Pass config object to oslo_reports https://review.openstack.org/485575 | |
| 16:07:16 | openstackgerrit | Merged openstack/nova master: Enhance support matrix document https://review.openstack.org/482020 | |
| 16:08:17 | openstackgerrit | Merged openstack/nova master: trivial: Remove some single use function from utils https://review.openstack.org/491513 | |
| 16:18:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Split flavors docs into admin and user guides https://review.openstack.org/501342 | |
| 16:20:23 | dtantsur | dansmith: thanks, I'll check it tomorrow - day full of errands today :( | |
| 16:20:32 | openstackgerrit | Dan Smith proposed openstack/nova master: Add nova-manage db command for ironic flavor migrations https://review.openstack.org/501025 | |
| 16:20:33 | openstackgerrit | Dan Smith proposed openstack/nova master: Add ComputeNodeList.get_by_hypervisor_type() https://review.openstack.org/501343 | |
| 16:20:33 | openstackgerrit | Dan Smith proposed openstack/nova master: Split out the core of the ironic flavor migration https://review.openstack.org/501024 | |
| 16:20:42 | dansmith | dtantsur: ack, np | |
| 16:21:20 | dtantsur | cdent: any specific schedule plans for API SIG PTG? | |
| 16:21:31 | cdent | dansmith: could you give me some early feedback on whether you think this is the right approach: https://review.openstack.org/#/c/501051/ (it’s related to post /allocations) | |
| 16:21:35 | dtantsur | cdent: I'm figuring out conflicts with https://etherpad.openstack.org/p/InteropDenver2017PTG | |
| 16:22:00 | dims | jaypipes : when is the hurricane expected? | |
| 16:22:03 | dansmith | cdent: yeah I know I need to do that, sorry | |
| 16:22:12 | dansmith | dims: weekend I think | |
| 16:22:59 | dansmith | dims: meanwhile portland is on fire and florida refuses to share the water: the | |
| 16:23:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/486660 | |
| 16:23:03 | dansmith | http://www.oregonlive.com/wildfires/index.ssf/2017/09/eagle_creek_fire_rages_in_colu.html | |
| 16:23:03 | cdent | dtantsur: edleafe and I talked about that yesterday and we’re going to try to cook a schedule, but as there aren’t specific topics (nobody signed up for guided reviews) that are really on the agenda, it is mostly going to be workshopping guideline improvements and whatever comes up | |
| 16:23:12 | cdent | no worries dansmith | |
| 16:23:14 | dims | dansmith : y i heard :) | |
| 16:23:42 | dtantsur | cdent: surprisingly to me, the guided review idea has not got too much traction with other ironic team members.. | |
| 16:23:49 | dims | dansmith : how far is it away from you? | |
| 16:24:21 | dansmith | dims: not far enough, but not currently a risk for us | |
| 16:24:40 | dansmith | dims: but it's crazy smoky outside and ash is falling on everything | |
| 16:25:15 | dansmith | walking outside is like getting a face full of camp fire | |
| 16:25:27 | dims | dansmith : dear lord | |
| 16:25:38 | dansmith | meanwhile, florida is all like "oh water, what are we going to do with all this water? did we mention we have a lot of water on the way?" | |
| 16:25:54 | cdent | that’s just rude | |
| 16:25:59 | dansmith | ikr | |
| 16:26:14 | mnaser | ahem im trying to troubleshoot (what i think is a nova bug).. any idea why instances can be seen in the list, but viewing them returns a 404? | |
| 16:26:32 | mnaser | ex: nova list shows them, nova show <uuid> says no server matching :X | |
| 16:26:41 | dansmith | mnaser: instance mapping missing? | |
| 16:26:45 | dims | dansmith : jaypipes : hopefully you will both make it to ptg | |
| 16:26:57 | dansmith | dims: thanks :) | |
| 16:27:24 | dansmith | mnaser: sounds like cell mappings are in place, so it can iterate and list, but no instance mapping for when it needs to go from instance uuid -> cell | |
| 16:27:35 | dansmith | mnaser: look in api database, instance_mappings table, see if there is one for the instance you're trying | |
| 16:27:38 | mnaser | i wonder why the mapping didnt go in then (customer says they recently failed builds) | |
| 16:28:29 | dansmith | mnaser: idk, should get an instance mapping once you're scheduled (or map you to cell0 if you fail) so we'd need to know what happened exactly to figure it out | |
| 16:28:32 | mnaser | /var/log/nova/nova-conductor.log:2017-09-06 10:42:20.512 829 WARNING nova.scheduler.utils [req-dd7ba87f-8539-4642-b238-3d1e4e4a686c 8873fec3d88b45c09af91ac942917650 eb60ffb2745a40e9b76223e5481e7c5e - default default] [instance: 90ecb4cc-8ea9-4419-9a3d-f67c9436df82] Setting instance to ERROR state.: TooManyInstances: Quota exceeded for ram: Requested 32768, but already used 716800 of 716800 ram | |
| 16:28:49 | mnaser | i wonder if it fails there meaning it doesn't end up getting scheduled? | |
| 16:28:58 | dansmith | mnaser: yeah so that should map you to cell0 | |
| 16:29:10 | mnaser | time to check if it did | |
| 16:29:36 | dansmith | mnaser: well, it seems like it's not mapped, but after confirming that, see if it's in the cell0 database | |
| 16:29:56 | mnaser | MariaDB [nova_cell0]> select * from instances where id='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => empty set | |
| 16:30:24 | dansmith | mnaser: uuid | |
| 16:30:29 | mnaser | oh duh | |
| 16:30:43 | mnaser | empty | |
| 16:30:52 | dansmith | check the other? | |
| 16:30:54 | mnaser | select * from instance_mappings where instance_uuid='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => cell_id is null | |
| 16:31:41 | mnaser | i can see it in my default clel db | |
| 16:31:57 | mnaser | (one that was created upgrading to ocata) | |
| 16:32:07 | mnaser | and the only one other than cell0 | |
| 16:32:18 | dansmith | okay that's weird | |
| 16:32:25 | dansmith | mnaser: can you file a bug with those details? | |
| 16:32:39 | mnaser | dansmith will do and ill do a bit more research too | |
| 16:32:42 | mnaser | maybe it happens when you hit quotas | |
| 16:32:52 | dansmith | mnaser: also, you should be able to do map_instances on your non-cell0 cell and fix that mapping I think | |
| 16:33:14 | dansmith | although you might have to nuke the instance mapping row first, which is also a bug for maintainabiity | |
| 16:33:18 | dansmith | mnaser: thanks | |
| 16:36:03 | openstackgerrit | Merged openstack/nova master: Glance download: only fsync files https://review.openstack.org/451094 | |
| 16:36:37 | openstackgerrit | Merged openstack/nova master: spelling mistake https://review.openstack.org/501215 | |
| 16:36:52 | mnaser | dansmith found the bug im pretty sure | |
| 16:37:00 | mnaser | https://github.com/openstack/nova/blob/a6c429bd7916211aa04d96caa54443ad92cb99ff/nova/conductor/manager.py#L992-L1006 | |
| 16:37:03 | mnaser | no bury in there | |
| 16:37:10 | mnaser | unlike the other failure cases | |
| 16:37:44 | mnaser | i guess i can work around it now by setting quota/recheck_quota=false for now.. but i think that's the failure case | |
| 16:47:26 | dansmith | mnaser: that's not the bug | |
| 16:47:40 | dansmith | mnaser: at that point we've created it in the one cell, so we can no longer bury it in cell0 | |
| 16:47:54 | dansmith | mnaser: so the bug is that we need to finish updating the mapping in that case, but apparently we aren't | |
| 16:47:59 | dansmith | mnaser: this is pike or ocata? | |
| 16:48:02 | mnaser | pike | |
| 16:48:33 | mnaser | this use case would launch like 30 instances at once | |
| 16:48:34 | dansmith | okay | |
| 16:48:41 | mnaser | s/user use case/ | |
| 16:48:56 | mnaser | and i guess it goes over quota in that big burst | |
| 16:49:09 | dansmith | mnaser: L1042 is the problem | |
| 16:49:24 | dansmith | meaning, we never run L1042, but we should | |
| 16:49:27 | dansmith | in that quota casw | |
| 16:49:39 | mnaser | so technically we should be saving before checking the quota again | |
| 16:49:55 | manasm | mriedem: did you get a chance to look at my comment regarding commit 1b2f3c7bb25667527e7e1d2afd84d5d5533e7751 earlier today? | |
| 16:50:04 | openstackgerrit | Merged openstack/nova master: Put base policy rules at first https://review.openstack.org/500736 | |
| 16:50:21 | mnaser | if folks are busy i dont mind picking up the work cause its a bit critical for us | |
| 16:50:28 | mnaser | just need a little pointing to what has to be done | |
| 16:51:56 | dansmith | mnaser: sure, I'll help guide when you're back to work on it. thanks for excellent triage | |
| 16:52:00 | manasm | the evacuated instance cleanup is failing in delete_allocation_for_evacuated_instance complaining that the compute node key is not present in resource_tracker compute_nodes. It looks like we need to ensure that resources have initialized (via _update_available_resource) before this is called? | |
| 16:53:12 | mriedem | mnaser: did you report a bug yet? | |
| 16:53:58 | mriedem | manasm: i didn't that no, but i think gibi has a related bug open | |
| 16:54:05 | mriedem | about a PUT /allocations with an empty body/ | |
| 16:54:06 | mriedem | ? | |