| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-06 | |||
| 16:26:41 | dansmith | mnaser: instance mapping missing? | |
| 16:26:45 | dims | dansmith : jaypipes : hopefully you will both make it to ptg | |
| 16:26:57 | dansmith | dims: thanks :) | |
| 16:27:24 | dansmith | mnaser: sounds like cell mappings are in place, so it can iterate and list, but no instance mapping for when it needs to go from instance uuid -> cell | |
| 16:27:35 | dansmith | mnaser: look in api database, instance_mappings table, see if there is one for the instance you're trying | |
| 16:27:38 | mnaser | i wonder why the mapping didnt go in then (customer says they recently failed builds) | |
| 16:28:29 | dansmith | mnaser: idk, should get an instance mapping once you're scheduled (or map you to cell0 if you fail) so we'd need to know what happened exactly to figure it out | |
| 16:28:32 | mnaser | /var/log/nova/nova-conductor.log:2017-09-06 10:42:20.512 829 WARNING nova.scheduler.utils [req-dd7ba87f-8539-4642-b238-3d1e4e4a686c 8873fec3d88b45c09af91ac942917650 eb60ffb2745a40e9b76223e5481e7c5e - default default] [instance: 90ecb4cc-8ea9-4419-9a3d-f67c9436df82] Setting instance to ERROR state.: TooManyInstances: Quota exceeded for ram: Requested 32768, but already used 716800 of 716800 ram | |
| 16:28:49 | mnaser | i wonder if it fails there meaning it doesn't end up getting scheduled? | |
| 16:28:58 | dansmith | mnaser: yeah so that should map you to cell0 | |
| 16:29:10 | mnaser | time to check if it did | |
| 16:29:36 | dansmith | mnaser: well, it seems like it's not mapped, but after confirming that, see if it's in the cell0 database | |
| 16:29:56 | mnaser | MariaDB [nova_cell0]> select * from instances where id='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => empty set | |
| 16:30:24 | dansmith | mnaser: uuid | |
| 16:30:29 | mnaser | oh duh | |
| 16:30:43 | mnaser | empty | |
| 16:30:52 | dansmith | check the other? | |
| 16:30:54 | mnaser | select * from instance_mappings where instance_uuid='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => cell_id is null | |
| 16:31:41 | mnaser | i can see it in my default clel db | |
| 16:31:57 | mnaser | (one that was created upgrading to ocata) | |
| 16:32:07 | mnaser | and the only one other than cell0 | |
| 16:32:18 | dansmith | okay that's weird | |
| 16:32:25 | dansmith | mnaser: can you file a bug with those details? | |
| 16:32:39 | mnaser | dansmith will do and ill do a bit more research too | |
| 16:32:42 | mnaser | maybe it happens when you hit quotas | |
| 16:32:52 | dansmith | mnaser: also, you should be able to do map_instances on your non-cell0 cell and fix that mapping I think | |
| 16:33:14 | dansmith | although you might have to nuke the instance mapping row first, which is also a bug for maintainabiity | |
| 16:33:18 | dansmith | mnaser: thanks | |
| 16:36:03 | openstackgerrit | Merged openstack/nova master: Glance download: only fsync files https://review.openstack.org/451094 | |
| 16:36:37 | openstackgerrit | Merged openstack/nova master: spelling mistake https://review.openstack.org/501215 | |
| 16:36:52 | mnaser | dansmith found the bug im pretty sure | |
| 16:37:00 | mnaser | https://github.com/openstack/nova/blob/a6c429bd7916211aa04d96caa54443ad92cb99ff/nova/conductor/manager.py#L992-L1006 | |
| 16:37:03 | mnaser | no bury in there | |
| 16:37:10 | mnaser | unlike the other failure cases | |
| 16:37:44 | mnaser | i guess i can work around it now by setting quota/recheck_quota=false for now.. but i think that's the failure case | |
| 16:47:26 | dansmith | mnaser: that's not the bug | |
| 16:47:40 | dansmith | mnaser: at that point we've created it in the one cell, so we can no longer bury it in cell0 | |
| 16:47:54 | dansmith | mnaser: so the bug is that we need to finish updating the mapping in that case, but apparently we aren't | |
| 16:47:59 | dansmith | mnaser: this is pike or ocata? | |
| 16:48:02 | mnaser | pike | |
| 16:48:33 | mnaser | this use case would launch like 30 instances at once | |
| 16:48:34 | dansmith | okay | |
| 16:48:41 | mnaser | s/user use case/ | |
| 16:48:56 | mnaser | and i guess it goes over quota in that big burst | |
| 16:49:09 | dansmith | mnaser: L1042 is the problem | |
| 16:49:24 | dansmith | meaning, we never run L1042, but we should | |
| 16:49:27 | dansmith | in that quota casw | |
| 16:49:39 | mnaser | so technically we should be saving before checking the quota again | |
| 16:49:55 | manasm | mriedem: did you get a chance to look at my comment regarding commit 1b2f3c7bb25667527e7e1d2afd84d5d5533e7751 earlier today? | |
| 16:50:04 | openstackgerrit | Merged openstack/nova master: Put base policy rules at first https://review.openstack.org/500736 | |
| 16:50:21 | mnaser | if folks are busy i dont mind picking up the work cause its a bit critical for us | |
| 16:50:28 | mnaser | just need a little pointing to what has to be done | |
| 16:51:56 | dansmith | mnaser: sure, I'll help guide when you're back to work on it. thanks for excellent triage | |
| 16:52:00 | manasm | the evacuated instance cleanup is failing in delete_allocation_for_evacuated_instance complaining that the compute node key is not present in resource_tracker compute_nodes. It looks like we need to ensure that resources have initialized (via _update_available_resource) before this is called? | |
| 16:53:12 | mriedem | mnaser: did you report a bug yet? | |
| 16:53:58 | mriedem | manasm: i didn't that no, but i think gibi has a related bug open | |
| 16:54:05 | mriedem | about a PUT /allocations with an empty body/ | |
| 16:54:06 | mriedem | ? | |
| 16:56:18 | manasm | mriedem: gibi: can you please help me with defect number so that I can confirm if its the same? | |
| 16:59:37 | mriedem | manasm: https://bugs.launchpad.net/nova/+bug/1714924 ? | |
| 16:59:38 | openstack | Launchpad bug 1714924 in OpenStack Compute (nova) "placement api fails when nova tries to delete resource allocation after failed evacuation" [Medium,Confirmed] | |
| 17:03:51 | openstackgerrit | Michael Still proposed openstack/nova master: Move nbd commands to privsep. https://review.openstack.org/500351 | |
| 17:03:51 | openstackgerrit | Michael Still proposed openstack/nova master: Move lvm handling to privsep. https://review.openstack.org/495516 | |
| 17:03:53 | openstackgerrit | Michael Still proposed openstack/nova master: Move xend existence probes to privsep. https://review.openstack.org/495538 | |
| 17:03:53 | openstackgerrit | Michael Still proposed openstack/nova master: Move shred to privsep. https://review.openstack.org/495537 | |
| 17:03:54 | openstackgerrit | Michael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls https://review.openstack.org/494423 | |
| 17:03:55 | openstackgerrit | Michael Still proposed openstack/nova master: WIP / Aspirational: we don't need rootwrap any more. https://review.openstack.org/495542 | |
| 17:03:56 | openstackgerrit | Michael Still proposed openstack/nova master: Move loopback setup and removal to privsep. https://review.openstack.org/495664 | |
| 17:04:00 | openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | |
| 17:04:02 | openstackgerrit | Michael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree() https://review.openstack.org/492326 | |
| 17:04:03 | openstackgerrit | Michael Still proposed openstack/nova master: Move the idmapshift binary into privsep. https://review.openstack.org/495541 | |
| 17:04:04 | openstackgerrit | Michael Still proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325 | |
| 17:04:07 | openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | |
| 17:04:08 | openstackgerrit | Michael Still proposed openstack/nova master: Move kpartx calls to privsep. https://review.openstack.org/500354 | |
| 17:04:13 | openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | |
| 17:04:19 | openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | |
| 17:04:22 | openstackgerrit | Michael Still proposed openstack/nova master: Move execs of touch to privsep. https://review.openstack.org/489190 | |
| 17:04:23 | openstackgerrit | Michael Still proposed openstack/nova master: Move libvirts dmcrypt support to privsep. https://review.openstack.org/490737 | |
| 17:04:35 | openstackgerrit | Michael Still proposed openstack/nova master: Move blkid calls to privsep. https://review.openstack.org/500398 | |
| 17:04:40 | openstackgerrit | Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438 | |
| 17:05:14 | sdague | oh, a wild mikal | |
| 17:05:21 | mikal | sdague: I _think_ I've addressed your concerns on https://review.openstack.org/#/c/459166/33, but your comments are welcome as always | |
| 17:07:34 | mikal | sdague: I know I've broken things later in the series with your requested changes, but I am going through and fixing those up now (it will take a while because of test runs) | |
| 17:08:31 | dansmith | sdague: what percentage of the remaining gate resources would you say mikal's set above consumes? :) | |
| 17:09:08 | sdague | dansmith: meh, I think we've been pegged all day already anyway | |
| 17:09:19 | sdague | mikal: the patch above that has an easy fix | |
| 17:09:26 | sdague | which is just delete the uml section | |
| 17:09:28 | dansmith | heh | |
| 17:10:54 | sdague | stephenfin: you deleted uml support at some point, right? | |
| 17:12:31 | sdague | dansmith: well it looks like zuul went super grumpy anyway and just merge conflicted the whole stack on itself | |
| 17:12:37 | sdague | so no tests being run | |
| 17:12:43 | dansmith | karma is a bitch | |
| 17:13:06 | kmalloc | sdague: I want to talk with you re some limit stuff being stored in keystone before the PTG session if you'll have a moment or two | |
| 17:14:10 | kmalloc | I want to make sure I'm not way off in the weeds. we can meet somewhere around the room(s), but just want to make sure i'm not thinking about this side-angled. | |
| 17:14:44 | sdague | kmalloc: sure, though I think the bigger question is finding someone else to drive the limits/quota work, because I definitely don't have the bw on it any more. | |
| 17:14:53 | sdague | but, yet, I'm happy to chat up current thinking | |
| 17:15:00 | kmalloc | right. i expect someone will have to help drive it | |
| 17:15:18 | kmalloc | but i know you and I spent a chunk of time going over the details and requirements :) | |
| 17:15:25 | kmalloc | i also don't have the bw to drive it | |
| 17:15:28 | kmalloc | =/ | |