| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 18:59:02 | superdan | without it they're 100% fail because they never have host mappings | |
| 18:59:16 | mriedem | even though, regardless of all those changes, | |
| 18:59:16 | mriedem | http://logs.openstack.org/15/428415/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/92a342e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-02-03_01_06_01_049 | |
| 18:59:24 | mriedem | Attempting claim: memory 4096 MB, disk 40 GB, vcpus 1 CPU | |
| 18:59:29 | openstackgerrit | John Perkins proposed openstack/nova master: Integrate oslo.config validator https://review.openstack.org/428974 | |
| 18:59:32 | mriedem | Total disk: 40 GB, used: 0.00 GB | |
| 18:59:39 | mriedem | so that instance claims all of the disk | |
| 18:59:46 | mriedem | which is why the next one fails | |
| 18:59:54 | mriedem | unless, there should be more nodes reporting in | |
| 19:00:09 | mriedem | i.e. we should have more than 40GB total | |
| 19:00:09 | superdan | or the scheduler raced | |
| 19:00:59 | cdent | whose 40GB is that? Is there a chance ironic is now trying to claim the wrong disk? (we are talking ironic here aren't we?) | |
| 19:01:18 | mriedem | yes ironic | |
| 19:03:02 | mriedem | 2017-02-03 01:05:33.326 28176 DEBUG nova.virt.ironic.driver [req-4a6f8f71-565c-4379-8288-347b8e9da028 - - - - -] Returning 3 available node(s) get_available_nodes /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:610 | |
| 19:03:26 | mriedem | Hypervisor/Node resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f free_ram=8192MB free_disk=40GB free_vcpus=4 pci_devices=None | |
| 19:03:54 | mriedem | Final resource view: name=7c2a1870-c2f8-46d5-b1a4-c72443bfc82f phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:05 | mriedem | Final resource view: name=aa97370e-02a2-4aa9-8833-616d30a4d589 phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:13 | mriedem | Final resource view: name=6a085a83-e051-480b-bf37-d092d7bbf13e phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 19:04:18 | mriedem | so those are the 3 nodes | |
| 19:04:22 | mriedem | 40GB each | |
| 19:04:36 | superdan | right and ironic doesn't sum the disks I think | |
| 19:04:51 | superdan | to prevent you from trying to schedule a 120G instance to that compute right/ | |
| 19:04:54 | mriedem | so 1 instance should be placed on 1 node right? | |
| 19:05:11 | superdan | yep | |
| 19:05:25 | superdan | I might be thinking of the summing in the api bit | |
| 19:05:31 | superdan | anyway, three nodes, 40g each | |
| 19:05:31 | mriedem | would be nice if the "Claim successful" message told us which node the resources were claimed on | |
| 19:05:43 | mriedem | in this log it's building a few instances | |
| 19:05:49 | mriedem | and it looks like one hits on a node that's used | |
| 19:06:15 | openstackgerrit | John Perkins proposed openstack/nova master: Integrate oslo.config validator https://review.openstack.org/428974 | |
| 19:06:16 | superdan | i.e. scheduler race | |
| 19:06:28 | superdan | aka that thing we say is explicitly out of scope: scheduling to 100% capacity | |
| 19:07:43 | openstackgerrit | Ed Leafe proposed openstack/nova master: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | |
| 19:10:48 | openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Update reno for stable/ocata https://review.openstack.org/428983 | |
| 19:12:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nodename to _claim_test log messages https://review.openstack.org/428985 | |
| 19:13:15 | cdent | figleaf, johnthetubaguy: Is https://review.openstack.org/#/c/404472/ no longer targetted for some version of ocata, or is it still under consideration? | |
| 19:14:22 | mriedem | cdent: i'm not considering it unless johnthetubaguy is super comfortable with it, | |
| 19:14:30 | mriedem | i've been out of the loop on that one since last week | |
| 19:14:44 | mriedem | as discussed a bit in here yesterday, i think we can do that in pike, | |
| 19:15:02 | mriedem | and put a service version on it so that the scheduler could know when it can start making requests with custom resource classes | |
| 19:15:07 | cdent | I think there are still some outstanding questions on it, maybe addressed by figleaf just now, so it does feel a bit late | |
| 19:15:10 | mriedem | in the ironic host manager scheduler thing | |
| 19:15:56 | johnthetubaguy | yeah, the questions don't all seem answered yet | |
| 19:16:43 | johnthetubaguy | I think we update the allocations more frequently than I first though, but I don't quite understand how the old allocations get refreshed | |
| 19:16:49 | johnthetubaguy | figleaf: ^ | |
| 19:17:27 | cfriesen | I've got an idea for an enhancement, wondering if it's worth writing up a spec. Basically the idea is that currently when booting the instance we allow "--user-data" and "--file", but when you rebuild the instance we only support "--file". So I'd like to propose adding support for "--user-data" when doing a rebuild. | |
| 19:17:34 | johnthetubaguy | it also seems odd to rebuild the inventory for the resource provider in two places | |
| 19:17:43 | cfriesen | Is this a dumb idea that'd get shot down immediately, or is it worth writing up? | |
| 19:18:44 | johnthetubaguy | cfriesen: seems worth adding a spec, make sense I think, config drive will be recreated either way, so why not I guess | |
| 19:19:20 | figleaf | johnthetubaguy: I'm not very clear on what part of this is confusing to you. I guess the fact that a lot of it is confusing is an issue in and of itself. | |
| 19:19:59 | openstackgerrit | Maciej Szankin proposed openstack/nova master: conf: deprecate key_manager.fixed _key opt https://review.openstack.org/428992 | |
| 19:20:27 | mriedem | cfriesen: didn't this come up recently-ish in the ML? | |
| 19:20:42 | mriedem | cfriesen: something about injected files not persisting across a rebuild? | |
| 19:20:44 | cdent | "I guess the fact that a lot of it is confusing is an issue in and of itself."++ | |
| 19:20:47 | mriedem | or evacuate maybe | |
| 19:21:14 | melwitt | yeah, I dunno what the reasons might be for not allowing it, but I have recalled mikal having the opinion that config drive is a once-at-init thing and not creating/changing it after instance boot | |
| 19:21:15 | mriedem | cfriesen: which was tied to a thread i had about deprecating/removing file injection | |
| 19:21:21 | melwitt | so I feel like he's the one to talk to | |
| 19:21:22 | cfriesen | mriedem: yes, userdata currently gets lost over a rebuild | |
| 19:21:44 | johnthetubaguy | figleaf: how do all the old instance allocations get refreshed, basically | |
| 19:21:44 | melwitt | it doesn't get lost, it stays on the config drive. or it should | |
| 19:21:47 | mriedem | i was thinking of the injected files / evacuate thing specifically i think | |
| 19:21:48 | cfriesen | mreidem: are you talking about removing --userdata from boot as well? | |
| 19:21:54 | johnthetubaguy | figleaf: I see how the new ones get the correct allocations | |
| 19:22:02 | mriedem | cfriesen: no | |
| 19:22:04 | johnthetubaguy | figleaf: I guess its possible when nova-compute first starts? | |
| 19:22:05 | mriedem | just --files | |
| 19:22:09 | mriedem | or personality or whatever | |
| 19:22:34 | mriedem | let me see if i can find the thread | |
| 19:22:56 | johnthetubaguy | mriedem: files will get lost, it only appears in the initial config drive, we don't store them | |
| 19:23:09 | mriedem | johnthetubaguy: yup i know | |
| 19:23:09 | cfriesen | mriedem: right. so currently we don't allow specifying files on an evacuate, but they are preserved | |
| 19:23:17 | mriedem | it was one of the reasons for killing file injection | |
| 19:23:32 | mriedem | cfriesen: http://lists.openstack.org/pipermail/openstack-dev/2016-November/107195.html | |
| 19:23:39 | cfriesen | mriedem: preserved in the config drive, right | |
| 19:24:04 | johnthetubaguy | cfriesen: well, depends if you are using shared storage I guess | |
| 19:24:05 | cfriesen | mriedem: yes | |
| 19:24:37 | cfriesen | mridem: so if we're still going to allow --user-data on boot, I think it'd make sense to allow it on rebuild | |
| 19:24:38 | figleaf | johnthetubaguy: the _init_compute_node is one place, yes | |
| 19:24:59 | johnthetubaguy | figleaf: because tracked instances is empty to start with, so everything is treated as a new instance? | |
| 19:25:03 | mriedem | cfriesen: yeah i think that's fine | |
| 19:25:10 | figleaf | johnthetubaguy: then self._update_usage_from_instances keeps them in sync | |
| 19:25:14 | melwitt | cfriesen: I'm still stuck on the part where you say userdata gets wiped out. it didn't use to on rebuild, it should just stay on the config drive? | |
| 19:25:42 | melwitt | this came up before at my last company and we tested it and userdata stayed intact across a rebuild | |
| 19:25:50 | figleaf | johnthetubaguy: yeah, that happens in _init_compute_node | |
| 19:25:52 | mriedem | 14:20:38 <johnthetubaguy> #action deprecate localfs file injection in ocata, remove in pike | |
| 19:25:52 | cfriesen | melwitt: currently injected files get lost on evacuate | |
| 19:25:53 | mriedem | damn! | |
| 19:26:07 | melwitt | hmm | |
| 19:26:15 | cfriesen | melwitt: currently userdata stays intact over a rebuild. my usecase is for people that want to rebuild to a different image with different userdata | |
| 19:26:40 | melwitt | cfriesen: yeah. okay, that makes sense. I thought earlier you said something about userdata getting wiped out during a rebuild | |
| 19:26:40 | cdent | figleaf, johnthetubaguy: tracked_instances is emptied every periodic job and _init_compute_node is called every periodic job | |
| 19:26:50 | superdan | cfriesen: isn't that called "boot" ? | |
| 19:27:03 | figleaf | cdent: oh geez - really? | |
| 19:27:10 | cdent | figleaf: you read this yet: http://lists.openstack.org/pipermail/openstack-dev/2017-January/110953.html | |
| 19:27:24 | cfriesen | superdan: you would think | |
| 19:27:30 | EmilienM | superdan: do you think I need to file a bug on tripleo side for the ironic thing? (wrt the patch in tripleo-common owalsh did) | |
| 19:27:42 | cfriesen | superdan: as I recall it ended up being related to heat updates | |
| 19:27:49 | figleaf | cdent: yes, but I don't have it committed to memory :) | |
| 19:28:03 | cdent | figleaf: oh, but you should | |