| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-06 | |||
| 15:30:59 | bauzas | mriedem: roger. just in a meeting now | |
| 15:31:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add release note for force live migration allocations https://review.openstack.org/501314 | |
| 15:31:33 | efried | esberglu Can you try setting `use_rootwrap_daemon` to True in the compute conf and see if the problem magically disappears? | |
| 15:31:57 | efried | esberglu That's in the [DEFAULT] section | |
| 15:32:01 | esberglu | efried: Yeah that isn't set currently. I can give it a try | |
| 15:32:23 | efried | esberglu Though if non-daemon is busted, that seems like it oughtta be a bug to nova. | |
| 15:34:05 | openstackgerrit | Merged openstack/nova master: Trim the fat from InstanceInfo https://review.openstack.org/471146 | |
| 15:34:31 | sdague | efried: yeh, though the real fix would be to get rid of it entirely for privsep | |
| 15:34:39 | sdague | be aware, that mikal's patch series is doing that | |
| 15:34:44 | efried | sdague "it" which? | |
| 15:34:58 | efried | The Process version? | |
| 15:35:13 | sdague | https://review.openstack.org/#/c/489438 | |
| 15:35:20 | sdague | that's 5 patches in | |
| 15:35:44 | sdague | and it will probably take some time to land, but rootwrap for tee is removed at that point | |
| 15:36:21 | sdague | efried: the tee call at all | |
| 15:37:18 | efried | sdague Mm. And the resolution is to decorate my function with this privsep gizmo and then do a regular ol `with open(): write()` ? | |
| 15:37:27 | sdague | yep | |
| 15:37:41 | sdague | but his first patch has to land before that will work | |
| 15:38:36 | efried | esberglu ^^ FYI. Need to add that to the watchlist so we can react when it lands. | |
| 15:39:54 | esberglu | efried: Will do | |
| 15:40:27 | efried | esberglu It would seem we could transition as soon as https://review.openstack.org/#/c/459166/ is in. | |
| 15:40:51 | efried | But in the meantime, <shrug> let's see if using the rootwrap daemon helps. | |
| 15:44:06 | stephenfin | jaypipes: https://review.openstack.org/#/c/498817/ and friends have their gate +1 now, if you were idle enough :) | |
| 15:56:19 | dansmith | mriedem: yeah I had marked that as -W ahead of me writing the release note, but then I wiped it when I pushed a little fix | |
| 15:57:00 | mriedem | i don't know how the tests are even passing | |
| 15:57:12 | dansmith | mriedem: because you missed that there is a patch before this? | |
| 15:57:41 | mriedem | yes | |
| 16:00:14 | esberglu | efried: ^ 459166 just got W+1 | |
| 16:00:59 | esberglu | efried: Oh wait nvm looking at the wrong thing | |
| 16:01:25 | efried | esberglu Ya know, we could patch that guy in and make our own patch that uses it... | |
| 16:05:33 | openstackgerrit | Merged openstack/nova master: Remove unneeded attributes from context https://review.openstack.org/498615 | |
| 16:06:14 | openstackgerrit | Merged openstack/nova master: Pass config object to oslo_reports https://review.openstack.org/485575 | |
| 16:07:16 | openstackgerrit | Merged openstack/nova master: Enhance support matrix document https://review.openstack.org/482020 | |
| 16:08:17 | openstackgerrit | Merged openstack/nova master: trivial: Remove some single use function from utils https://review.openstack.org/491513 | |
| 16:18:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Split flavors docs into admin and user guides https://review.openstack.org/501342 | |
| 16:20:23 | dtantsur | dansmith: thanks, I'll check it tomorrow - day full of errands today :( | |
| 16:20:32 | openstackgerrit | Dan Smith proposed openstack/nova master: Add nova-manage db command for ironic flavor migrations https://review.openstack.org/501025 | |
| 16:20:33 | openstackgerrit | Dan Smith proposed openstack/nova master: Add ComputeNodeList.get_by_hypervisor_type() https://review.openstack.org/501343 | |
| 16:20:33 | openstackgerrit | Dan Smith proposed openstack/nova master: Split out the core of the ironic flavor migration https://review.openstack.org/501024 | |
| 16:20:42 | dansmith | dtantsur: ack, np | |
| 16:21:20 | dtantsur | cdent: any specific schedule plans for API SIG PTG? | |
| 16:21:31 | cdent | dansmith: could you give me some early feedback on whether you think this is the right approach: https://review.openstack.org/#/c/501051/ (it’s related to post /allocations) | |
| 16:21:35 | dtantsur | cdent: I'm figuring out conflicts with https://etherpad.openstack.org/p/InteropDenver2017PTG | |
| 16:22:00 | dims | jaypipes : when is the hurricane expected? | |
| 16:22:03 | dansmith | cdent: yeah I know I need to do that, sorry | |
| 16:22:12 | dansmith | dims: weekend I think | |
| 16:22:59 | dansmith | dims: meanwhile portland is on fire and florida refuses to share the water: the | |
| 16:23:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/486660 | |
| 16:23:03 | dansmith | http://www.oregonlive.com/wildfires/index.ssf/2017/09/eagle_creek_fire_rages_in_colu.html | |
| 16:23:03 | cdent | dtantsur: edleafe and I talked about that yesterday and we’re going to try to cook a schedule, but as there aren’t specific topics (nobody signed up for guided reviews) that are really on the agenda, it is mostly going to be workshopping guideline improvements and whatever comes up | |
| 16:23:12 | cdent | no worries dansmith | |
| 16:23:14 | dims | dansmith : y i heard :) | |
| 16:23:42 | dtantsur | cdent: surprisingly to me, the guided review idea has not got too much traction with other ironic team members.. | |
| 16:23:49 | dims | dansmith : how far is it away from you? | |
| 16:24:21 | dansmith | dims: not far enough, but not currently a risk for us | |
| 16:24:40 | dansmith | dims: but it's crazy smoky outside and ash is falling on everything | |
| 16:25:15 | dansmith | walking outside is like getting a face full of camp fire | |
| 16:25:27 | dims | dansmith : dear lord | |
| 16:25:38 | dansmith | meanwhile, florida is all like "oh water, what are we going to do with all this water? did we mention we have a lot of water on the way?" | |
| 16:25:54 | cdent | that’s just rude | |
| 16:25:59 | dansmith | ikr | |
| 16:26:14 | mnaser | ahem im trying to troubleshoot (what i think is a nova bug).. any idea why instances can be seen in the list, but viewing them returns a 404? | |
| 16:26:32 | mnaser | ex: nova list shows them, nova show <uuid> says no server matching :X | |
| 16:26:41 | dansmith | mnaser: instance mapping missing? | |
| 16:26:45 | dims | dansmith : jaypipes : hopefully you will both make it to ptg | |
| 16:26:57 | dansmith | dims: thanks :) | |
| 16:27:24 | dansmith | mnaser: sounds like cell mappings are in place, so it can iterate and list, but no instance mapping for when it needs to go from instance uuid -> cell | |
| 16:27:35 | dansmith | mnaser: look in api database, instance_mappings table, see if there is one for the instance you're trying | |
| 16:27:38 | mnaser | i wonder why the mapping didnt go in then (customer says they recently failed builds) | |
| 16:28:29 | dansmith | mnaser: idk, should get an instance mapping once you're scheduled (or map you to cell0 if you fail) so we'd need to know what happened exactly to figure it out | |
| 16:28:32 | mnaser | /var/log/nova/nova-conductor.log:2017-09-06 10:42:20.512 829 WARNING nova.scheduler.utils [req-dd7ba87f-8539-4642-b238-3d1e4e4a686c 8873fec3d88b45c09af91ac942917650 eb60ffb2745a40e9b76223e5481e7c5e - default default] [instance: 90ecb4cc-8ea9-4419-9a3d-f67c9436df82] Setting instance to ERROR state.: TooManyInstances: Quota exceeded for ram: Requested 32768, but already used 716800 of 716800 ram | |
| 16:28:49 | mnaser | i wonder if it fails there meaning it doesn't end up getting scheduled? | |
| 16:28:58 | dansmith | mnaser: yeah so that should map you to cell0 | |
| 16:29:10 | mnaser | time to check if it did | |
| 16:29:36 | dansmith | mnaser: well, it seems like it's not mapped, but after confirming that, see if it's in the cell0 database | |
| 16:29:56 | mnaser | MariaDB [nova_cell0]> select * from instances where id='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => empty set | |
| 16:30:24 | dansmith | mnaser: uuid | |
| 16:30:29 | mnaser | oh duh | |
| 16:30:43 | mnaser | empty | |
| 16:30:52 | dansmith | check the other? | |
| 16:30:54 | mnaser | select * from instance_mappings where instance_uuid='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => cell_id is null | |
| 16:31:41 | mnaser | i can see it in my default clel db | |
| 16:31:57 | mnaser | (one that was created upgrading to ocata) | |
| 16:32:07 | mnaser | and the only one other than cell0 | |
| 16:32:18 | dansmith | okay that's weird | |
| 16:32:25 | dansmith | mnaser: can you file a bug with those details? | |
| 16:32:39 | mnaser | dansmith will do and ill do a bit more research too | |
| 16:32:42 | mnaser | maybe it happens when you hit quotas | |
| 16:32:52 | dansmith | mnaser: also, you should be able to do map_instances on your non-cell0 cell and fix that mapping I think | |
| 16:33:14 | dansmith | although you might have to nuke the instance mapping row first, which is also a bug for maintainabiity | |
| 16:33:18 | dansmith | mnaser: thanks | |
| 16:36:03 | openstackgerrit | Merged openstack/nova master: Glance download: only fsync files https://review.openstack.org/451094 | |
| 16:36:37 | openstackgerrit | Merged openstack/nova master: spelling mistake https://review.openstack.org/501215 | |
| 16:36:52 | mnaser | dansmith found the bug im pretty sure | |
| 16:37:00 | mnaser | https://github.com/openstack/nova/blob/a6c429bd7916211aa04d96caa54443ad92cb99ff/nova/conductor/manager.py#L992-L1006 | |
| 16:37:03 | mnaser | no bury in there | |
| 16:37:10 | mnaser | unlike the other failure cases | |
| 16:37:44 | mnaser | i guess i can work around it now by setting quota/recheck_quota=false for now.. but i think that's the failure case | |
| 16:47:26 | dansmith | mnaser: that's not the bug | |
| 16:47:40 | dansmith | mnaser: at that point we've created it in the one cell, so we can no longer bury it in cell0 | |