| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-14 | |||
| 21:19:33 | eandersson | I am trying to figure out why the Server Group Tests are failing because the resource usage isn't always updated. | |
| 21:19:56 | eandersson | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L168 | |
| 21:20:27 | eandersson | Need to figure out if it is the test that is broken, or if there is a race condition happening here. | |
| 21:20:45 | eandersson | jaypipes maybe? | |
| 21:30:44 | mriedem | sdague: any idea why we'd cache compute nodes in the request here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L110 ? | |
| 21:30:53 | mriedem | nothing is looking up a cached compute node record from what i can tell | |
| 21:30:55 | mriedem | anywhere else in the api | |
| 21:31:01 | mriedem | do we just do it for fun and games? | |
| 21:38:13 | oomichi | mriedem: I guessed that was a reason why unknown queries should be blocked, but now additionalProperties is True on https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/keypairs.py#L94 | |
| 21:38:30 | mriedem | yeah i looked up the original change that added that and it wasn't clear to me | |
| 21:38:45 | oomichi | mriedem: so I can not find the reason now. alex_xu can answer anyways | |
| 21:38:55 | mriedem | yeah i have a lot of questions for alex_xu | |
| 21:39:12 | mriedem | 5:40am for him, i'm not sure why he's not up and working yet :) | |
| 21:39:13 | oomichi | oh, "a lot".. | |
| 21:39:50 | mriedem | see john asked about the same sort of thing here https://review.openstack.org/#/c/398755/18/nova/api/openstack/compute/schemas/keypairs.py | |
| 21:40:03 | mriedem | btw, i removed those decorators from the methods that didn't need them and no tests failed | |
| 21:40:08 | mriedem | so i'm not really sure why we need them | |
| 21:40:11 | mriedem | but that's a question for alex | |
| 21:40:16 | mriedem | one of many | |
| 21:40:58 | oomichi | hehe, I saw the patch on the history. and I found I +2d on that before without this question | |
| 21:41:14 | mriedem | oomichi: did you see my question to sean above about https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L110 ? | |
| 21:41:24 | mriedem | as far as i can tell, nothing uses those cached compute nodes in the request | |
| 21:41:32 | mriedem | i think the cache is more or less for extensions isn't it? | |
| 21:43:39 | melwitt | that's my understanding, that those request caches are just to avoid doing DB lookup every time for each extension that receives the request | |
| 21:44:30 | melwitt | since multiple extensions can operate on the same request | |
| 21:44:59 | mriedem | i don't see anything extending the "hypervisors" resource extension | |
| 21:45:09 | mriedem | therefore i don't think anything would be using the cache | |
| 21:45:18 | mriedem | maybe something used to and we collapsed it? | |
| 21:45:26 | mriedem | unless the view does somehow | |
| 21:45:37 | melwitt | they don't have to extend it to be run, I don't think | |
| 21:45:52 | mriedem | https://review.openstack.org/#/c/53785/ | |
| 21:46:02 | mriedem | aha | |
| 21:46:04 | mriedem | os-pci | |
| 21:46:05 | mriedem | mfer | |
| 21:46:10 | melwitt | can't remember for sure though. I just know a lot of it is like onion layers, running each extension on the request object | |
| 21:46:20 | mriedem | time to get out my delete knife | |
| 22:04:50 | melwitt | eandersson: it looks like there might be something wrong with the test. in the log trace there's "Compute_service record created for compute:fake-mini" but then later on there's "Received an update from an unknown host 'compute'. Re-created its InstanceList." which looks like it's not matching it to the earlier created compute | |
| 22:06:05 | melwitt | like I wonder if the first compute service in the test needs to be started more similarly as the second compute service with the host and nodes set for it | |
| 22:08:20 | eandersson | Yea - found that as well | |
| 22:08:37 | jaypipes | eandersson: agree with melwitt. I don't really see anything in that patch that should be producing that error. and it seems the failure doesn't always occur, thus the rechecks. | |
| 22:09:08 | jaypipes | eandersson: the whole disabled check should definitely not be coming into play here. | |
| 22:09:15 | eandersson | Testing it locally it occurs maybe 80% of the time though | |
| 22:09:21 | jaypipes | hmm | |
| 22:09:31 | jaypipes | and only those affinity tests? | |
| 22:09:41 | jaypipes | eandersson: ^ | |
| 22:10:10 | eandersson | I am just doing a single test at the moment (test_boot_servers_with_affinity_no_valid_host) | |
| 22:10:18 | eandersson | Adding a sleep at the right place and it succeeds 100% | |
| 22:11:48 | melwitt | eandersson: you might try patching like this and see if it improves anything locally http://paste.openstack.org/show/612599/ | |
| 22:12:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 22:12:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Handle uuids in os-hypervisors API https://review.openstack.org/471899 | |
| 22:12:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use uuid for id in os-services API https://review.openstack.org/464280 | |
| 22:12:19 | mriedem | melwitt: dansmith: oomichi: check out https://review.openstack.org/471899 if you want to see some terrible things | |
| 22:17:01 | cfriesen | mriedem: does nova-conductor need to be running in order to run "nova db online_data_migrations"? | |
| 22:17:47 | oomichi | mriedem: I am not sure why we want to add empty schema again for this api v2.0, is that WIP reason? | |
| 22:18:08 | oomichi | oh, these schemas are not used on https://review.openstack.org/#/c/471899/8/nova/api/openstack/compute/hypervisors.py .. | |
| 22:18:12 | eandersson | melwitt, was looking into something similar, still something holding it up | |
| 22:18:37 | eandersson | *but | |
| 22:18:59 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.48: Standardization of VM diagnostics https://review.openstack.org/473228 | |
| 22:19:08 | eandersson | got 100% on that one test now, lets see if the others are going to be as happy :D | |
| 22:19:34 | mriedem | oomichi: it's one of my questions to alex_xu | |
| 22:19:40 | mriedem | i don't want to use empty schema if it can be avoided | |
| 22:19:45 | mriedem | but yeah it's all WIP right now | |
| 22:19:59 | mriedem | cfriesen: that's a dansmith question | |
| 22:20:04 | mriedem | i know some of them go through the objects, | |
| 22:20:13 | mriedem | but i figure they should be local to the db | |
| 22:30:33 | eandersson | https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py#L97 | |
| 22:32:49 | eandersson | Looks like a very similar issue | |
| 22:35:43 | melwitt | huh, I added that. barely remember it | |
| 22:37:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 22:37:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Handle uuids in os-hypervisors API https://review.openstack.org/471899 | |
| 22:37:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use uuid for id in os-services API https://review.openstack.org/464280 | |
| 22:37:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Centralize compute_node_search_by_hypervisor in os-hypervisors https://review.openstack.org/474381 | |
| 22:37:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop caching compute nodes in the request https://review.openstack.org/474382 | |
| 22:37:11 | mriedem | easy pickins on those bottom two changes | |
| 22:40:14 | eandersson | mriedem, so your suggestion fixed most issues, but it's still hitting this in 4 tests > not self.driver.node_is_available(nodename) | |
| 22:40:25 | openstackgerrit | Merged openstack/nova master: Count server group members to check quota https://review.openstack.org/457859 | |
| 22:42:18 | eandersson | I see - FAKE_NODES need to have both hosts | |
| 22:48:28 | melwitt | mriedem: just curious, how was the compute node request cache messing up what you're working on? | |
| 22:49:16 | openstackgerrit | Merged openstack/nova master: Make compute_node_statistics() work across cells https://review.openstack.org/473931 | |
| 22:53:23 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Re-worked ram filter and weights to support overcommit https://review.openstack.org/390984 | |
| 22:53:40 | eandersson | That was effort, but I think I got it figured out | |
| 22:54:29 | eandersson | I could probably have left fake_network where it was though | |
| 22:59:22 | eandersson | thanks melwitt | |
| 23:02:53 | melwitt | eandersson: np. you might probably be asked to split the test fix into its own patch ahead of your scheduler patch. the issue in the server group tests is probably affecting other things too and is its own separate bug | |
| 23:03:18 | eandersson | Yea - was thinking the same. | |
| 23:04:15 | eandersson | Gonna go and get luncch and Zuul is hopefully all green :D delayed lunch by like... 4 hours trying to figure this out :D | |
| 23:04:27 | eandersson | *by the time I get back | |
| #openstack-nova - 2017-06-15 | |||
| 00:06:28 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Re-worked ram filter and weights to support overcommit https://review.openstack.org/390984 | |
| 00:08:31 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Re-worked ram filter and weights to support overcommit https://review.openstack.org/390984 | |
| 00:11:59 | mriedem | melwitt: it wasn't | |
| 00:12:11 | mriedem | melwitt: but the amount of gorp in the hypervisors api controller code hurts me | |
| 00:12:12 | mriedem | inside | |
| 00:12:40 | mriedem | so before i add more complication in there, i'm trying to trim the fat a bit | |
| 00:32:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: cleanup PUT /os-hypervisors/statistics docs https://review.openstack.org/474250 | |
| 00:37:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop caching compute nodes in the request https://review.openstack.org/474382 | |
| 00:37:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 00:37:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Centralize compute_node_search_by_hypervisor in os-hypervisors https://review.openstack.org/474381 | |
| 00:37:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Handle uuids in os-hypervisors API https://review.openstack.org/471899 | |
| 00:37:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use uuid for id in os-services API https://review.openstack.org/464280 | |
| 01:03:08 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Re-worked ram filter and weights to support overcommit https://review.openstack.org/390984 | |