Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-06
16:22:12 dansmith dims: weekend I think
16:22:59 dansmith dims: meanwhile portland is on fire and florida refuses to share the water: the
16:23:00 openstackgerrit Stephen Finucane proposed openstack/nova master: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/486660
16:23:03 cdent dtantsur: edleafe and I talked about that yesterday and we’re going to try to cook a schedule, but as there aren’t specific topics (nobody signed up for guided reviews) that are really on the agenda, it is mostly going to be workshopping guideline improvements and whatever comes up
16:23:03 dansmith http://www.oregonlive.com/wildfires/index.ssf/2017/09/eagle_creek_fire_rages_in_colu.html
16:23:12 cdent no worries dansmith
16:23:14 dims dansmith : y i heard :)
16:23:42 dtantsur cdent: surprisingly to me, the guided review idea has not got too much traction with other ironic team members..
16:23:49 dims dansmith : how far is it away from you?
16:24:21 dansmith dims: not far enough, but not currently a risk for us
16:24:40 dansmith dims: but it's crazy smoky outside and ash is falling on everything
16:25:15 dansmith walking outside is like getting a face full of camp fire
16:25:27 dims dansmith : dear lord
16:25:38 dansmith meanwhile, florida is all like "oh water, what are we going to do with all this water? did we mention we have a lot of water on the way?"
16:25:54 cdent that’s just rude
16:25:59 dansmith ikr
16:26:14 mnaser ahem im trying to troubleshoot (what i think is a nova bug).. any idea why instances can be seen in the list, but viewing them returns a 404?
16:26:32 mnaser ex: nova list shows them, nova show <uuid> says no server matching :X
16:26:41 dansmith mnaser: instance mapping missing?
16:26:45 dims dansmith : jaypipes : hopefully you will both make it to ptg
16:26:57 dansmith dims: thanks :)
16:27:24 dansmith mnaser: sounds like cell mappings are in place, so it can iterate and list, but no instance mapping for when it needs to go from instance uuid -> cell
16:27:35 dansmith mnaser: look in api database, instance_mappings table, see if there is one for the instance you're trying
16:27:38 mnaser i wonder why the mapping didnt go in then (customer says they recently failed builds)
16:28:29 dansmith mnaser: idk, should get an instance mapping once you're scheduled (or map you to cell0 if you fail) so we'd need to know what happened exactly to figure it out
16:28:32 mnaser /var/log/nova/nova-conductor.log:2017-09-06 10:42:20.512 829 WARNING nova.scheduler.utils [req-dd7ba87f-8539-4642-b238-3d1e4e4a686c 8873fec3d88b45c09af91ac942917650 eb60ffb2745a40e9b76223e5481e7c5e - default default] [instance: 90ecb4cc-8ea9-4419-9a3d-f67c9436df82] Setting instance to ERROR state.: TooManyInstances: Quota exceeded for ram: Requested 32768, but already used 716800 of 716800 ram
16:28:49 mnaser i wonder if it fails there meaning it doesn't end up getting scheduled?
16:28:58 dansmith mnaser: yeah so that should map you to cell0
16:29:10 mnaser time to check if it did
16:29:36 dansmith mnaser: well, it seems like it's not mapped, but after confirming that, see if it's in the cell0 database
16:29:56 mnaser MariaDB [nova_cell0]> select * from instances where id='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => empty set
16:30:24 dansmith mnaser: uuid
16:30:29 mnaser oh duh
16:30:43 mnaser empty
16:30:52 dansmith check the other?
16:30:54 mnaser select * from instance_mappings where instance_uuid='90ecb4cc-8ea9-4419-9a3d-f67c9436df82'; => cell_id is null
16:31:41 mnaser i can see it in my default clel db
16:31:57 mnaser (one that was created upgrading to ocata)
16:32:07 mnaser and the only one other than cell0
16:32:18 dansmith okay that's weird
16:32:25 dansmith mnaser: can you file a bug with those details?
16:32:39 mnaser dansmith will do and ill do a bit more research too
16:32:42 mnaser maybe it happens when you hit quotas
16:32:52 dansmith mnaser: also, you should be able to do map_instances on your non-cell0 cell and fix that mapping I think
16:33:14 dansmith although you might have to nuke the instance mapping row first, which is also a bug for maintainabiity
16:33:18 dansmith mnaser: thanks
16:36:03 openstackgerrit Merged openstack/nova master: Glance download: only fsync files https://review.openstack.org/451094
16:36:37 openstackgerrit Merged openstack/nova master: spelling mistake https://review.openstack.org/501215
16:36:52 mnaser dansmith found the bug im pretty sure
16:37:00 mnaser https://github.com/openstack/nova/blob/a6c429bd7916211aa04d96caa54443ad92cb99ff/nova/conductor/manager.py#L992-L1006
16:37:03 mnaser no bury in there
16:37:10 mnaser unlike the other failure cases
16:37:44 mnaser i guess i can work around it now by setting quota/recheck_quota=false for now.. but i think that's the failure case
16:47:26 dansmith mnaser: that's not the bug
16:47:40 dansmith mnaser: at that point we've created it in the one cell, so we can no longer bury it in cell0
16:47:54 dansmith mnaser: so the bug is that we need to finish updating the mapping in that case, but apparently we aren't
16:47:59 dansmith mnaser: this is pike or ocata?
16:48:02 mnaser pike
16:48:33 mnaser this use case would launch like 30 instances at once
16:48:34 dansmith okay
16:48:41 mnaser s/user use case/
16:48:56 mnaser and i guess it goes over quota in that big burst
16:49:09 dansmith mnaser: L1042 is the problem
16:49:24 dansmith meaning, we never run L1042, but we should
16:49:27 dansmith in that quota casw
16:49:39 mnaser so technically we should be saving before checking the quota again
16:49:55 manasm mriedem: did you get a chance to look at my comment regarding commit 1b2f3c7bb25667527e7e1d2afd84d5d5533e7751 earlier today?
16:50:04 openstackgerrit Merged openstack/nova master: Put base policy rules at first https://review.openstack.org/500736
16:50:21 mnaser if folks are busy i dont mind picking up the work cause its a bit critical for us
16:50:28 mnaser just need a little pointing to what has to be done
16:51:56 dansmith mnaser: sure, I'll help guide when you're back to work on it. thanks for excellent triage
16:52:00 manasm the evacuated instance cleanup is failing in delete_allocation_for_evacuated_instance complaining that the compute node key is not present in resource_tracker compute_nodes. It looks like we need to ensure that resources have initialized (via _update_available_resource) before this is called?
16:53:12 mriedem mnaser: did you report a bug yet?
16:53:58 mriedem manasm: i didn't that no, but i think gibi has a related bug open
16:54:05 mriedem about a PUT /allocations with an empty body/
16:54:06 mriedem ?
16:56:18 manasm mriedem: gibi: can you please help me with defect number so that I can confirm if its the same?
16:59:37 mriedem manasm: https://bugs.launchpad.net/nova/+bug/1714924 ?
16:59:38 openstack Launchpad bug 1714924 in OpenStack Compute (nova) "placement api fails when nova tries to delete resource allocation after failed evacuation" [Medium,Confirmed]
17:03:51 openstackgerrit Michael Still proposed openstack/nova master: Move nbd commands to privsep. https://review.openstack.org/500351
17:03:51 openstackgerrit Michael Still proposed openstack/nova master: Move lvm handling to privsep. https://review.openstack.org/495516
17:03:53 openstackgerrit Michael Still proposed openstack/nova master: Move xend existence probes to privsep. https://review.openstack.org/495538
17:03:53 openstackgerrit Michael Still proposed openstack/nova master: Move shred to privsep. https://review.openstack.org/495537
17:03:54 openstackgerrit Michael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls https://review.openstack.org/494423
17:03:55 openstackgerrit Michael Still proposed openstack/nova master: WIP / Aspirational: we don't need rootwrap any more. https://review.openstack.org/495542
17:03:56 openstackgerrit Michael Still proposed openstack/nova master: Move loopback setup and removal to privsep. https://review.openstack.org/495664
17:04:00 openstackgerrit Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972
17:04:02 openstackgerrit Michael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree() https://review.openstack.org/492326
17:04:03 openstackgerrit Michael Still proposed openstack/nova master: Move the idmapshift binary into privsep. https://review.openstack.org/495541
17:04:04 openstackgerrit Michael Still proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325
17:04:07 openstackgerrit Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486
17:04:08 openstackgerrit Michael Still proposed openstack/nova master: Move kpartx calls to privsep. https://review.openstack.org/500354
17:04:13 openstackgerrit Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229
17:04:19 openstackgerrit Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166
17:04:22 openstackgerrit Michael Still proposed openstack/nova master: Move execs of touch to privsep. https://review.openstack.org/489190
17:04:23 openstackgerrit Michael Still proposed openstack/nova master: Move libvirts dmcrypt support to privsep. https://review.openstack.org/490737
17:04:35 openstackgerrit Michael Still proposed openstack/nova master: Move blkid calls to privsep. https://review.openstack.org/500398
17:04:40 openstackgerrit Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438
17:05:14 sdague oh, a wild mikal
17:05:21 mikal sdague: I _think_ I've addressed your concerns on https://review.openstack.org/#/c/459166/33, but your comments are welcome as always

Earlier   Later