| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-20 | |||
| 14:05:17 | jroll | I guess you're trying to boot an instance after it's created anyway | |
| 14:05:21 | jaypipes | mriedem: yes, and looking at the placement log, I see 3 PUTs for creating inventory successfully. | |
| 14:05:37 | jroll | EmilienM: what does your flavor look like? | |
| 14:05:44 | EmilienM | I can try a revert and Depends-On tripleo patch | |
| 14:05:50 | EmilienM | so I can tell if this patch broke us or not | |
| 14:06:01 | EmilienM | jroll: let me show you that | |
| 14:06:04 | EmilienM | jroll: gimme 2 min | |
| 14:06:15 | jroll | that would be good data, yeah | |
| 14:06:17 | jroll | EmilienM: thanks | |
| 14:07:47 | openstackgerrit | Emilien Macchi proposed openstack/nova master: DNM - Revert "virt: implement get_inventory() for Ironic" https://review.openstack.org/447512 | |
| 14:07:51 | mriedem | Using cache for node eca19329-b67f-46b5-8509-586491a15f17, age: 7.65818786621 _node_from_cache /usr/lib/python2.7/site-packages/nova/virt/ironic/driver.py:700 | |
| 14:08:06 | mriedem | is there a timer on the node cache in the ironic driver? | |
| 14:08:14 | jroll | it refreshes every resource tracker run | |
| 14:09:29 | mriedem | jroll: are you sure about that? | |
| 14:09:37 | mriedem | i don't see _node_from_cache refreshing | |
| 14:09:50 | mriedem | and get_available_resource only does if it doesn't exist | |
| 14:10:00 | jroll | mriedem: get_available_nodes | |
| 14:10:21 | jroll | ya https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L604 | |
| 14:11:39 | EmilienM | jroll: I think the default flavor is 4096, 1, 40 | |
| 14:11:53 | mriedem | and get_available_nodes is called for every update_available_resource in the compute manager, which is on startup and every 60s | |
| 14:11:56 | jaypipes | mriedem: get_available_resource() is always called before get_inventory(), and get_available_resource() always refreshes the node cache: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L688 | |
| 14:12:07 | EmilienM | jroll: details here: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-api.txt.gz#_2017-03-19_06_38_20_764 | |
| 14:12:21 | mriedem | jaypipes: get_available_resource does not always refresh the cache | |
| 14:12:21 | openstackgerrit | Rafael Folco proposed openstack/nova master: DELETE all inventory for a resource provider https://review.openstack.org/416669 | |
| 14:12:26 | mriedem | see the conditional block it's in | |
| 14:12:30 | mriedem | if not self.node_cache: | |
| 14:12:34 | mriedem | self._refresh_cache() | |
| 14:12:54 | jroll | EmilienM: okay, and we have Final resource view: name=eca19329-b67f-46b5-8509-586491a15f17 phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 14:12:55 | jaypipes | mriedem: well, I mean it will always trigger a full pull of the node list if the node cache is empty. | |
| 14:13:00 | mriedem | anyway, looks like jroll is right and we refresh in get_available_nodes, which is called in each periodic in the manager | |
| 14:13:51 | dansmith | jroll: no, but what has changed? | |
| 14:14:16 | jaypipes | jroll: placement will return nodes that have enough RAM/VCPU/DISK | |
| 14:14:19 | jroll | dansmith: we now have ironic inventory in the placement engine | |
| 14:14:31 | jroll | right, that's what I thought, so this flavor should fit | |
| 14:14:32 | jaypipes | jroll: scheduler isn't yet requesting custom resource class | |
| 14:14:35 | dansmith | jroll: sure but we're not asking for it | |
| 14:15:14 | jroll | the scheduler isn't asking placement for resource providers that match the flavor's cpu/ram/disk? | |
| 14:15:22 | jroll | oh wait, that was already there | |
| 14:15:23 | jroll | blah | |
| 14:15:28 | dansmith | right | |
| 14:15:46 | mriedem | Hypervisor: VCPU information unavailable _report_hypervisor_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:676 | |
| 14:15:53 | mriedem | indicates the node is not available, right ^ ? | |
| 14:16:40 | jroll | mriedem: yeah, but later it figures things out: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-03-19_07_14_43_029 | |
| 14:16:51 | EmilienM | (in the meantime, we're testing the revert here: https://review.openstack.org/#/c/447514) | |
| 14:17:03 | jroll | log level filters here would be super useful, btw :P | |
| 14:17:34 | dansmith | jroll: yeah :( | |
| 14:17:41 | EmilienM | jroll: it's in our (long) todo | |
| 14:17:48 | dansmith | jaypipes: a scheduling decision looks like /resource_providers?stuff=stuff right? | |
| 14:17:49 | EmilienM | but yeah, I agree | |
| 14:17:51 | dansmith | jaypipes: in placement | |
| 14:17:58 | jaypipes | dansmith: yep. | |
| 14:18:05 | dansmith | jaypipes: I don't see a single one of those in placement's log | |
| 14:18:15 | jroll | I do | |
| 14:18:22 | jroll | GET /placement/resource_providers?resources=DISK_GB%3A40%2CMEMORY_MB%3A4096%2CVCPU%3A1 | |
| 14:18:23 | dansmith | I think we should be asking placement, then using that list of uuids to filter the compute_nodes list right? | |
| 14:18:37 | dansmith | jroll: ah, yeah, okay | |
| 14:18:47 | jaypipes | EmilienM: jroll is referring to better logging in the scheduler/placement service. | |
| 14:18:48 | dansmith | must have fat-fingered my search | |
| 14:19:09 | jroll | jaypipes: no, I'm referring to being able to filter compute logs by log level in tripleo logs :) | |
| 14:19:10 | mriedem | jaypipes: i think it's per-job | |
| 14:19:13 | mriedem | right | |
| 14:19:16 | mriedem | we have filtering on the dsvm jbos | |
| 14:19:17 | EmilienM | jaypipes: ah ok. I was talking about the capacity of filtering logs like devstack does | |
| 14:19:18 | mriedem | *jobs | |
| 14:19:21 | jaypipes | dansmith: right, but placement is returning 0 providers. | |
| 14:19:30 | dansmith | jaypipes: is it? | |
| 14:19:49 | jroll | looks like it to me | |
| 14:19:56 | jroll | "Starting with 0 host(s) get_filtered_objects" | |
| 14:19:59 | dansmith | okay that's different than no compute node records | |
| 14:20:02 | dansmith | jroll: no, that's too late to know | |
| 14:20:08 | jroll | is it? | |
| 14:20:10 | jroll | lol | |
| 14:20:11 | jaypipes | dansmith: "Starting with 0 hosts" | |
| 14:20:23 | dansmith | jaypipes: that's after we've correlated the compute nodes and the placement result | |
| 14:21:03 | jroll | ah, yeah | |
| 14:21:34 | dansmith | I wanna know if placement is returning zero, or we have no CNs in the DB | |
| 14:21:38 | dansmith | I have been assuming the latter | |
| 14:21:46 | dansmith | since we see us reporting from the compute node to placemetn | |
| 14:21:48 | jroll | ya, needs more logging | |
| 14:22:59 | EmilienM | is the log level of tripleo jobs is enough? let me know if I can help | |
| 14:23:23 | jroll | EmilienM: it's a failure to log things in nova code, not the log level | |
| 14:23:25 | jaypipes | EmilienM: yeah, it is I think. it's just we need more debug logging in nova-scheduler. | |
| 14:23:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Log placement result in scheduler https://review.openstack.org/447520 | |
| 14:23:32 | dansmith | this ^ | |
| 14:23:44 | jaypipes | thx dansmith | |
| 14:23:51 | jroll | ++ | |
| 14:24:13 | dansmith | EmilienM: can you make something run on top of that log patch? | |
| 14:24:26 | mriedem | that's going to blow up | |
| 14:24:53 | mriedem | that's just a debug change right? are we planning on merging that? | |
| 14:24:59 | dansmith | mriedem: no, just for debug | |
| 14:25:17 | dansmith | mriedem: the real one need not log rps in the warning case I think | |
| 14:25:18 | dansmith | I just wanted to know if it's empty or none | |
| 14:25:19 | mriedem | please DNM prefix the commit message or something, or -2 | |
| 14:25:21 | jroll | can we merge it just so mriedem gets slammed with logs? | |
| 14:25:38 | mriedem | dansmith: agree, but i could see someone mistakenly approving this | |
| 14:25:56 | EmilienM | dansmith: yes | |
| 14:26:00 | EmilienM | my revert | |
| 14:26:04 | EmilienM | oh wait | |
| 14:26:06 | EmilienM | no | |
| 14:26:11 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Log placement result in scheduler https://review.openstack.org/447520 | |