| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 18:52:42 | mriedem | Total disk: 40 GB, used: 40.00 GB | |
| 18:52:44 | mriedem | right above that | |
| 18:52:48 | EmilienM | could it be nova/wsgi again? | |
| 18:53:02 | mriedem | and just before that Total disk: 40 GB, used: 0.00 GB | |
| 18:53:05 | superdan | mriedem: unless nova-compute starts before any ironic nodes are provisioned | |
| 18:53:19 | superdan | mriedem: also, they stop reporting resources for nodes that are in the middle of a change | |
| 18:53:29 | superdan | mriedem: so they will go in and out and in and out over the course of a normal lifetime | |
| 18:53:32 | superdan | it's weird | |
| 18:53:34 | mriedem | there is an instance created right before that one that claims all the disk http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_052 | |
| 18:53:34 | EmilienM | superdan: in the case opposite happens, nova-ocmpute is able to update nodes resources? | |
| 18:53:40 | superdan | but jroll and I were looking at that the other day | |
| 18:54:20 | EmilienM | let me look in logstash if tripleo ci had this error before | |
| 18:54:43 | mriedem | EmilienM: logstash is super backed up | |
| 18:54:46 | mriedem | so probably no results there | |
| 18:54:46 | jroll | I didn't do it | |
| 18:55:29 | mriedem | plus and http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_02_020 is debug level which we don't index | |
| 18:56:05 | EmilienM | mriedem: over the last 30 days, one occurence, on 427536 changfe | |
| 18:56:15 | mriedem | we only store 10 days of logs | |
| 18:56:20 | mriedem | and it's been busted for several days | |
| 18:56:22 | EmilienM | mriedem: I think it's pretty clear this patch is responsible | |
| 18:56:26 | EmilienM | even 10 days | |
| 18:56:54 | mriedem | well the patch is https://review.openstack.org/#/c/427536/ | |
| 18:56:58 | EmilienM | mriedem: http://tripleo.org/cistatus.html we run so many jobs per day, we would have seen it before | |
| 18:56:59 | mriedem | 2 deps | |
| 18:57:03 | mriedem | i don't see how https://review.openstack.org/#/c/427782/ would cause this | |
| 18:57:11 | mriedem | so what does https://review.openstack.org/#/c/427534/ do? | |
| 18:57:21 | mriedem | my money is on ^ | |
| 18:57:28 | mriedem | "Ensure that the ironic nodes have been picked up by the nova resource tracker before running nova-manage cell_v2 host discovery." | |
| 18:57:46 | EmilienM | owalsh: you around? ^ | |
| 18:57:47 | superdan | 427782 is only about getting instances via the api, really nothing to do with any of that | |
| 18:57:53 | mriedem | right | |
| 18:58:07 | mriedem | the latter is specifically about when to discover ironic hosts for resource tracking | |
| 18:58:24 | superdan | mriedem: right, and it is important because they weren't enrolling nodes by the time they ran discover | |
| 18:59:02 | superdan | without it they're 100% fail because they never have host mappings | |
| 18:59:16 | mriedem | http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_049 | |
| 18:59:16 | mriedem | even though, regardless of all those changes, | |
| 18:59:24 | mriedem | Attempting claim: memory 4096 MB, disk 40 GB, vcpus 1 CPU | |
| 18:59:29 | openstackgerrit | John Perkins proposed openstack/nova master: Integrate oslo.config validator https://review.openstack.org/428974 | |
| 18:59:32 | mriedem | Total disk: 40 GB, used: 0.00 GB | |
| 18:59:39 | mriedem | so that instance claims all of the disk | |
| 18:59:46 | mriedem | which is why the next one fails | |
| 18:59:54 | mriedem | unless, there should be more nodes reporting in | |
| 19:00:09 | superdan | or the scheduler raced | |
| 19:00:09 | mriedem | i.e. we should have more than 40GB total | |
| 19:00:59 | cdent | whose 40GB is that? Is there a chance ironic is now trying to claim the wrong disk? (we are talking ironic here aren't we?) | |
| 19:01:18 | mriedem | yes ironic | |
| 19:03:02 | mriedem | 2017-02-03 01:05:33.326 28176 DEBUG nova.virt.ironic.driver [req-4a6f8f71-565c-4379-8288-347b8e9da028 - - - - -] Returning 3 available node(s) get_available_nodes /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:610 | |
| 19:03:26 | mriedem | Hypervisor/Node resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f free_ram=8192MB free_disk=40GB free_vcpus=4 pci_devices=None | |
| 19:03:54 | mriedem | Final resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:05 | mriedem | Final resource view: name=aa97370e-02a2-4aa9-8833-616d30a4d589 phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:13 | mriedem | Final resource view: name=6a085a83-e051-480b-bf37-d092d7bbf13e phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:18 | mriedem | so those are the 3 nodes | |
| 19:04:22 | mriedem | 40GB each | |
| 19:04:36 | superdan | right and ironic doesn't sum the disks I think | |
| 19:04:51 | superdan | to prevent you from trying to schedule a 120G instance to that compute right/ | |
| 19:04:54 | mriedem | so 1 instance should be placed on 1 node right? | |
| 19:05:11 | superdan | yep | |
| 19:05:25 | superdan | I might be thinking of the summing in the api bit | |
| 19:05:31 | mriedem | would be nice if the "Claim successful" message told us which node the resources were claimed on | |
| 19:05:31 | superdan | anyway, three nodes, 40g each | |
| 19:05:43 | mriedem | in this log it's building a few instances | |
| 19:05:49 | mriedem | and it looks like one hits on a node that's used | |
| 19:06:15 | openstackgerrit | John Perkins proposed openstack/nova master: Integrate oslo.config validator https://review.openstack.org/428974 | |
| 19:06:16 | superdan | i.e. scheduler race | |
| 19:06:28 | superdan | aka that thing we say is explicitly out of scope: scheduling to 100% capacity | |
| 19:07:43 | openstackgerrit | Ed Leafe proposed openstack/nova master: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | |
| 19:10:48 | openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Update reno for stable/ocata https://review.openstack.org/428983 | |
| 19:12:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nodename to _claim_test log messages https://review.openstack.org/428985 | |
| 19:13:15 | cdent | figleaf, johnthetubaguy: Is https://review.openstack.org/#/c/404472/ no longer targetted for some version of ocata, or is it still under consideration? | |
| 19:14:22 | mriedem | cdent: i'm not considering it unless johnthetubaguy is super comfortable with it, | |
| 19:14:30 | mriedem | i've been out of the loop on that one since last week | |
| 19:14:44 | mriedem | as discussed a bit in here yesterday, i think we can do that in pike, | |
| 19:15:02 | mriedem | and put a service version on it so that the scheduler could know when it can start making requests with custom resource classes | |
| 19:15:07 | cdent | I think there are still some outstanding questions on it, maybe addressed by figleaf just now, so it does feel a bit late | |
| 19:15:10 | mriedem | in the ironic host manager scheduler thing | |
| 19:15:56 | johnthetubaguy | yeah, the questions don't all seem answered yet | |
| 19:16:43 | johnthetubaguy | I think we update the allocations more frequently than I first though, but I don't quite understand how the old allocations get refreshed | |
| 19:16:49 | johnthetubaguy | figleaf: ^ | |
| 19:17:27 | cfriesen | I've got an idea for an enhancement, wondering if it's worth writing up a spec. Basically the idea is that currently when booting the instance we allow "--user-data" and "--file", but when you rebuild the instance we only support "--file". So I'd like to propose adding support for "--user-data" when doing a rebuild. | |
| 19:17:34 | johnthetubaguy | it also seems odd to rebuild the inventory for the resource provider in two places | |
| 19:17:43 | cfriesen | Is this a dumb idea that'd get shot down immediately, or is it worth writing up? | |
| 19:18:44 | johnthetubaguy | cfriesen: seems worth adding a spec, make sense I think, config drive will be recreated either way, so why not I guess | |
| 19:19:20 | figleaf | johnthetubaguy: I'm not very clear on what part of this is confusing to you. I guess the fact that a lot of it is confusing is an issue in and of itself. | |
| 19:19:59 | openstackgerrit | Maciej Szankin proposed openstack/nova master: conf: deprecate key_manager.fixed _key opt https://review.openstack.org/428992 | |
| 19:20:27 | mriedem | cfriesen: didn't this come up recently-ish in the ML? | |
| 19:20:42 | mriedem | cfriesen: something about injected files not persisting across a rebuild? | |
| 19:20:44 | cdent | "I guess the fact that a lot of it is confusing is an issue in and of itself."++ | |
| 19:20:47 | mriedem | or evacuate maybe | |
| 19:21:14 | melwitt | yeah, I dunno what the reasons might be for not allowing it, but I have recalled mikal having the opinion that config drive is a once-at-init thing and not creating/changing it after instance boot | |
| 19:21:15 | mriedem | cfriesen: which was tied to a thread i had about deprecating/removing file injection | |
| 19:21:21 | melwitt | so I feel like he's the one to talk to | |
| 19:21:22 | cfriesen | mriedem: yes, userdata currently gets lost over a rebuild | |
| 19:21:44 | melwitt | it doesn't get lost, it stays on the config drive. or it should | |
| 19:21:44 | johnthetubaguy | figleaf: how do all the old instance allocations get refreshed, basically | |
| 19:21:47 | mriedem | i was thinking of the injected files / evacuate thing specifically i think | |
| 19:21:48 | cfriesen | mreidem: are you talking about removing --userdata from boot as well? | |
| 19:21:54 | johnthetubaguy | figleaf: I see how the new ones get the correct allocations | |
| 19:22:02 | mriedem | cfriesen: no | |
| 19:22:04 | johnthetubaguy | figleaf: I guess its possible when nova-compute first starts? | |
| 19:22:05 | mriedem | just --files | |