| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-20 | |||
| 14:10:21 | jroll | ya https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L604 | |
| 14:11:39 | EmilienM | jroll: I think the default flavor is 4096, 1, 40 | |
| 14:11:53 | mriedem | and get_available_nodes is called for every update_available_resource in the compute manager, which is on startup and every 60s | |
| 14:11:56 | jaypipes | mriedem: get_available_resource() is always called before get_inventory(), and get_available_resource() always refreshes the node cache: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L688 | |
| 14:12:07 | EmilienM | jroll: details here: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-api.txt.gz#_2017-03-19_06_38_20_764 | |
| 14:12:21 | openstackgerrit | Rafael Folco proposed openstack/nova master: DELETE all inventory for a resource provider https://review.openstack.org/416669 | |
| 14:12:21 | mriedem | jaypipes: get_available_resource does not always refresh the cache | |
| 14:12:26 | mriedem | see the conditional block it's in | |
| 14:12:30 | mriedem | if not self.node_cache: | |
| 14:12:34 | mriedem | self._refresh_cache() | |
| 14:12:54 | jroll | EmilienM: okay, and we have Final resource view: name=eca19329-b67f-46b5-8509-586491a15f17 phys_ram=8192MB used_ram=0MB phys_disk=40GB used_disk=0GB total_vcpus=4 used_vcpus=0 pci_stats=[] | |
| 14:12:55 | jaypipes | mriedem: well, I mean it will always trigger a full pull of the node list if the node cache is empty. | |
| 14:13:00 | mriedem | anyway, looks like jroll is right and we refresh in get_available_nodes, which is called in each periodic in the manager | |
| 14:13:51 | dansmith | jroll: no, but what has changed? | |
| 14:14:16 | jaypipes | jroll: placement will return nodes that have enough RAM/VCPU/DISK | |
| 14:14:19 | jroll | dansmith: we now have ironic inventory in the placement engine | |
| 14:14:31 | jroll | right, that's what I thought, so this flavor should fit | |
| 14:14:32 | jaypipes | jroll: scheduler isn't yet requesting custom resource class | |
| 14:14:35 | dansmith | jroll: sure but we're not asking for it | |
| 14:15:14 | jroll | the scheduler isn't asking placement for resource providers that match the flavor's cpu/ram/disk? | |
| 14:15:22 | jroll | oh wait, that was already there | |
| 14:15:23 | jroll | blah | |
| 14:15:28 | dansmith | right | |
| 14:15:46 | mriedem | Hypervisor: VCPU information unavailable _report_hypervisor_resource_view /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:676 | |
| 14:15:53 | mriedem | indicates the node is not available, right ^ ? | |
| 14:16:40 | jroll | mriedem: yeah, but later it figures things out: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-03-19_07_14_43_029 | |
| 14:16:51 | EmilienM | (in the meantime, we're testing the revert here: https://review.openstack.org/#/c/447514) | |
| 14:17:03 | jroll | log level filters here would be super useful, btw :P | |
| 14:17:34 | dansmith | jroll: yeah :( | |
| 14:17:41 | EmilienM | jroll: it's in our (long) todo | |
| 14:17:48 | dansmith | jaypipes: a scheduling decision looks like /resource_providers?stuff=stuff right? | |
| 14:17:49 | EmilienM | but yeah, I agree | |
| 14:17:51 | dansmith | jaypipes: in placement | |
| 14:17:58 | jaypipes | dansmith: yep. | |
| 14:18:05 | dansmith | jaypipes: I don't see a single one of those in placement's log | |
| 14:18:15 | jroll | I do | |
| 14:18:22 | jroll | GET /placement/resource_providers?resources=DISK_GB%3A40%2CMEMORY_MB%3A4096%2CVCPU%3A1 | |
| 14:18:23 | dansmith | I think we should be asking placement, then using that list of uuids to filter the compute_nodes list right? | |
| 14:18:37 | dansmith | jroll: ah, yeah, okay | |
| 14:18:47 | jaypipes | EmilienM: jroll is referring to better logging in the scheduler/placement service. | |
| 14:18:48 | dansmith | must have fat-fingered my search | |
| 14:19:09 | jroll | jaypipes: no, I'm referring to being able to filter compute logs by log level in tripleo logs :) | |
| 14:19:10 | mriedem | jaypipes: i think it's per-job | |
| 14:19:13 | mriedem | right | |
| 14:19:16 | mriedem | we have filtering on the dsvm jbos | |
| 14:19:17 | EmilienM | jaypipes: ah ok. I was talking about the capacity of filtering logs like devstack does | |
| 14:19:18 | mriedem | *jobs | |
| 14:19:21 | jaypipes | dansmith: right, but placement is returning 0 providers. | |
| 14:19:30 | dansmith | jaypipes: is it? | |
| 14:19:49 | jroll | looks like it to me | |
| 14:19:56 | jroll | "Starting with 0 host(s) get_filtered_objects" | |
| 14:19:59 | dansmith | okay that's different than no compute node records | |
| 14:20:02 | dansmith | jroll: no, that's too late to know | |
| 14:20:08 | jroll | is it? | |
| 14:20:10 | jroll | lol | |
| 14:20:11 | jaypipes | dansmith: "Starting with 0 hosts" | |
| 14:20:23 | dansmith | jaypipes: that's after we've correlated the compute nodes and the placement result | |
| 14:21:03 | jroll | ah, yeah | |
| 14:21:34 | dansmith | I wanna know if placement is returning zero, or we have no CNs in the DB | |
| 14:21:38 | dansmith | I have been assuming the latter | |
| 14:21:46 | dansmith | since we see us reporting from the compute node to placemetn | |
| 14:21:48 | jroll | ya, needs more logging | |
| 14:22:59 | EmilienM | is the log level of tripleo jobs is enough? let me know if I can help | |
| 14:23:23 | jroll | EmilienM: it's a failure to log things in nova code, not the log level | |
| 14:23:25 | jaypipes | EmilienM: yeah, it is I think. it's just we need more debug logging in nova-scheduler. | |
| 14:23:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Log placement result in scheduler https://review.openstack.org/447520 | |
| 14:23:32 | dansmith | this ^ | |
| 14:23:44 | jaypipes | thx dansmith | |
| 14:23:51 | jroll | ++ | |
| 14:24:13 | dansmith | EmilienM: can you make something run on top of that log patch? | |
| 14:24:26 | mriedem | that's going to blow up | |
| 14:24:53 | mriedem | that's just a debug change right? are we planning on merging that? | |
| 14:24:59 | dansmith | mriedem: no, just for debug | |
| 14:25:17 | dansmith | mriedem: the real one need not log rps in the warning case I think | |
| 14:25:18 | dansmith | I just wanted to know if it's empty or none | |
| 14:25:19 | mriedem | please DNM prefix the commit message or something, or -2 | |
| 14:25:21 | jroll | can we merge it just so mriedem gets slammed with logs? | |
| 14:25:38 | mriedem | dansmith: agree, but i could see someone mistakenly approving this | |
| 14:25:56 | EmilienM | dansmith: yes | |
| 14:26:00 | EmilienM | my revert | |
| 14:26:04 | EmilienM | oh wait | |
| 14:26:06 | EmilienM | no | |
| 14:26:11 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Log placement result in scheduler https://review.openstack.org/447520 | |
| 14:26:12 | EmilienM | gimme a sec | |
| 14:28:02 | EmilienM | dansmith: here: https://review.openstack.org/#/c/447522/ | |
| 14:31:27 | dansmith | jroll: yeah, | |
| 14:31:29 | openstackgerrit | Lee Yarwood proposed openstack/nova master: encryptors: Switch to os-brick encryptor classes https://review.openstack.org/391597 | |
| 14:31:40 | efried | So it turns out that assert_called_once and assert_not_called aren't actually methods on a mock. Without speccing, they'll succeed happily just like any nonexistent method would. There's 80ish instances of these in the nova tree. May I open a bug and fix this, or is there something I don't know? | |
| 14:31:41 | dansmith | I don't even see some debug logs that we should be seeing in scheduler | |
| 14:31:59 | jroll | the only thing I've noticed so far is that the flavor isn't an exact match to the node | |
| 14:32:01 | jroll | oh, hm | |
| 14:32:03 | dansmith | like we should see this: https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L590-L591 | |
| 14:32:12 | dansmith | but unless I just suck at searching I can't find it | |
| 14:32:34 | jroll | agree | |
| 14:33:11 | mriedem | efried: are you looking at std lib or the mock library? | |
| 14:33:36 | efried | mriedem How do I know which is which? | |
| 14:33:40 | mriedem | efried: b/c they are in the lib https://github.com/testing-cabal/mock/blob/2.0.0/mock/mock.py#L890 | |
| 14:33:41 | mriedem | and we use the lib | |
| 14:33:44 | mriedem | in the py27 job | |
| 14:34:00 | mriedem | https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L149 | |