Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-06
16:23:12 cdent no worries dansmith
16:23:14 dims dansmith : y i heard :)
16:23:42 dtantsur cdent: surprisingly to me, the guided review idea has not got too much traction with other ironic team members..
16:23:49 dims dansmith : how far is it away from you?
16:24:21 dansmith dims: not far enough, but not currently a risk for us
16:24:40 dansmith dims: but it's crazy smoky outside and ash is falling on everything
16:25:15 dansmith walking outside is like getting a face full of camp fire
16:25:27 dims dansmith : dear lord
16:25:38 dansmith meanwhile, florida is all like "oh water, what are we going to do with all this water? did we mention we have a lot of water on the way?"
16:25:54 cdent that’s just rude
16:25:59 dansmith ikr
16:26:14 mnaser ahem im trying to troubleshoot (what i think is a nova bug).. any idea why instances can be seen in the list, but viewing them returns a 404?
16:26:32 mnaser ex: nova list shows them, nova show <uuid> says no server matching :X
16:26:41 dansmith mnaser: instance mapping missing?
16:26:45 dims dansmith : jaypipes : hopefully you will both make it to ptg
16:26:57 dansmith dims: thanks :)
16:27:24 dansmith mnaser: sounds like cell mappings are in place, so it can iterate and list, but no instance mapping for when it needs to go from instance uuid -> cell
16:27:35 dansmith mnaser: look in api database, instance_mappings table, see if there is one for the instance you're trying
16:27:38 mnaser i wonder why the mapping didnt go in then (customer says they recently failed builds)
16:28:29 dansmith mnaser: idk, should get an instance mapping once you're scheduled (or map you to cell0 if you fail) so we'd need to know what happened exactly to figure it out
16:28:32 mnaser /var/log/nova/nova-conductor.log:2017-09-06 10:42:20.512 829 WARNING nova.scheduler.utils [req-dd7ba87f-8539-4642-b238-3d1e4e4a686c 8873fec3d88b45c09af91ac942917650 eb60ffb2745a40e9b76223e5481e7c5e - default default] [instance: 90ecb4cc-8ea9-4419-9a3d-f67c9436df82] Setting instance to ERROR state.: TooManyInstances: Quota exceeded for ram: Requested 32768, but already used 716800 of 716800 ram
16:28:49 mnaser i wonder if it fails there meaning it doesn't end up getting scheduled?
16:28:58 dansmith mnaser: yeah so that should map you to cell0
16:29:10 mnaser time to check if it did
16:29:36 dansmith mnaser: well, it seems like it's not mapped, but after confirming that, see if it's in the cell0 database
16:29:56 mnaser MariaDB [nova_cell0]> select * from instances where id='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => empty set
16:30:24 dansmith mnaser: uuid
16:30:29 mnaser oh duh
16:30:43 mnaser empty
16:30:52 dansmith check the other?
16:30:54 mnaser select * from instance_mappings where instance_uuid='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => cell_id is null
16:31:41 mnaser i can see it in my default clel db
16:31:57 mnaser (one that was created upgrading to ocata)
16:32:07 mnaser and the only one other than cell0
16:32:18 dansmith okay that's weird
16:32:25 dansmith mnaser: can you file a bug with those details?
16:32:39 mnaser dansmith will do and ill do a bit more research too
16:32:42 mnaser maybe it happens when you hit quotas
16:32:52 dansmith mnaser: also, you should be able to do map_instances on your non-cell0 cell and fix that mapping I think
16:33:14 dansmith although you might have to nuke the instance mapping row first, which is also a bug for maintainabiity
16:33:18 dansmith mnaser: thanks
16:36:03 openstackgerrit Merged openstack/nova master: Glance download: only fsync files https://review.openstack.org/451094
16:36:37 openstackgerrit Merged openstack/nova master: spelling mistake https://review.openstack.org/501215
16:36:52 mnaser dansmith found the bug im pretty sure
16:37:00 mnaser https://github.com/openstack/nova/blob/a6c429bd7916211aa04d96caa54443ad92cb99ff/nova/conductor/manager.py#L992-L1006
16:37:03 mnaser no bury in there
16:37:10 mnaser unlike the other failure cases
16:37:44 mnaser i guess i can work around it now by setting quota/recheck_quota=false for now.. but i think that's the failure case
16:47:26 dansmith mnaser: that's not the bug
16:47:40 dansmith mnaser: at that point we've created it in the one cell, so we can no longer bury it in cell0
16:47:54 dansmith mnaser: so the bug is that we need to finish updating the mapping in that case, but apparently we aren't
16:47:59 dansmith mnaser: this is pike or ocata?
16:48:02 mnaser pike
16:48:33 mnaser this use case would launch like 30 instances at once
16:48:34 dansmith okay
16:48:41 mnaser s/user use case/
16:48:56 mnaser and i guess it goes over quota in that big burst
16:49:09 dansmith mnaser: L1042 is the problem
16:49:24 dansmith meaning, we never run L1042, but we should
16:49:27 dansmith in that quota casw
16:49:39 mnaser so technically we should be saving before checking the quota again
16:49:55 manasm mriedem: did you get a chance to look at my comment regarding commit 1b2f3c7bb25667527e7e1d2afd84d5d5533e7751 earlier today?
16:50:04 openstackgerrit Merged openstack/nova master: Put base policy rules at first https://review.openstack.org/500736
16:50:21 mnaser if folks are busy i dont mind picking up the work cause its a bit critical for us
16:50:28 mnaser just need a little pointing to what has to be done
16:51:56 dansmith mnaser: sure, I'll help guide when you're back to work on it. thanks for excellent triage
16:52:00 manasm the evacuated instance cleanup is failing in delete_allocation_for_evacuated_instance complaining that the compute node key is not present in resource_tracker compute_nodes. It looks like we need to ensure that resources have initialized (via _update_available_resource) before this is called?
16:53:12 mriedem mnaser: did you report a bug yet?
16:53:58 mriedem manasm: i didn't that no, but i think gibi has a related bug open
16:54:05 mriedem about a PUT /allocations with an empty body/
16:54:06 mriedem ?
16:56:18 manasm mriedem: gibi: can you please help me with defect number so that I can confirm if its the same?
16:59:37 mriedem manasm: https://bugs.launchpad.net/nova/+bug/1714924 ?
16:59:38 openstack Launchpad bug 1714924 in OpenStack Compute (nova) "placement api fails when nova tries to delete resource allocation after failed evacuation" [Medium,Confirmed]
17:03:51 openstackgerrit Michael Still proposed openstack/nova master: Move nbd commands to privsep. https://review.openstack.org/500351
17:03:51 openstackgerrit Michael Still proposed openstack/nova master: Move lvm handling to privsep. https://review.openstack.org/495516
17:03:53 openstackgerrit Michael Still proposed openstack/nova master: Move xend existence probes to privsep. https://review.openstack.org/495538
17:03:53 openstackgerrit Michael Still proposed openstack/nova master: Move shred to privsep. https://review.openstack.org/495537
17:03:54 openstackgerrit Michael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls https://review.openstack.org/494423
17:03:55 openstackgerrit Michael Still proposed openstack/nova master: WIP / Aspirational: we don't need rootwrap any more. https://review.openstack.org/495542
17:03:56 openstackgerrit Michael Still proposed openstack/nova master: Move loopback setup and removal to privsep. https://review.openstack.org/495664
17:04:00 openstackgerrit Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972
17:04:02 openstackgerrit Michael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree() https://review.openstack.org/492326
17:04:03 openstackgerrit Michael Still proposed openstack/nova master: Move the idmapshift binary into privsep. https://review.openstack.org/495541
17:04:04 openstackgerrit Michael Still proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325
17:04:07 openstackgerrit Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486
17:04:08 openstackgerrit Michael Still proposed openstack/nova master: Move kpartx calls to privsep. https://review.openstack.org/500354
17:04:13 openstackgerrit Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229
17:04:19 openstackgerrit Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166
17:04:22 openstackgerrit Michael Still proposed openstack/nova master: Move execs of touch to privsep. https://review.openstack.org/489190
17:04:23 openstackgerrit Michael Still proposed openstack/nova master: Move libvirts dmcrypt support to privsep. https://review.openstack.org/490737
17:04:35 openstackgerrit Michael Still proposed openstack/nova master: Move blkid calls to privsep. https://review.openstack.org/500398
17:04:40 openstackgerrit Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438
17:05:14 sdague oh, a wild mikal
17:05:21 mikal sdague: I _think_ I've addressed your concerns on https://review.openstack.org/#/c/459166/33, but your comments are welcome as always
17:07:34 mikal sdague: I know I've broken things later in the series with your requested changes, but I am going through and fixing those up now (it will take a while because of test runs)
17:08:31 dansmith sdague: what percentage of the remaining gate resources would you say mikal's set above consumes? :)
17:09:08 sdague dansmith: meh, I think we've been pegged all day already anyway
17:09:19 sdague mikal: the patch above that has an easy fix
17:09:26 sdague which is just delete the uml section

Earlier   Later