| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-27 | |||
| 09:55:58 | johnthetubaguy | looks like we get the wrong min version? | |
| 09:56:03 | bauzas | johnthetubaguy: exactly | |
| 09:56:08 | bauzas | we get 16 as minimum | |
| 09:56:16 | bauzas | so we call the placement API | |
| 09:56:27 | johnthetubaguy | super interesting if thats broken | |
| 09:56:34 | bauzas | but we should have 15 given we have an old subnode | |
| 09:57:07 | bauzas | what I'd love is some way to introspect more | |
| 09:57:19 | bauzas | like LOG.debugging the DB call itself | |
| 09:57:40 | johnthetubaguy | bauzas: we could add a line in there that lists all the nova compute service records | |
| 09:57:42 | bauzas | I can throw a patch but I have very limited access | |
| 09:57:55 | johnthetubaguy | I can do something if that fails | |
| 09:58:01 | johnthetubaguy | I assume we are still trying to get this in ASAP | |
| 09:58:11 | johnthetubaguy | its kinda release critical in my book | |
| 09:58:12 | bauzas | yup, mriedem said he granted a FFE | |
| 09:58:19 | johnthetubaguy | cool | |
| 09:58:27 | johnthetubaguy | I guess this is the only one so far right? | |
| 09:58:36 | bauzas | yeah, my one-day trip to Paris is terrible :/ | |
| 09:58:47 | johnthetubaguy | it seems a safe date to pick | |
| 09:58:51 | bauzas | mriedem said he'd love also custom-rc | |
| 09:58:53 | johnthetubaguy | that rest day after FFEE | |
| 09:59:01 | johnthetubaguy | oh, not been tracking those patches | |
| 09:59:07 | johnthetubaguy | that would be good, for iroinc | |
| 09:59:54 | bauzas | yup | |
| 09:59:59 | bauzas | tbh, me neither | |
| 10:00:38 | johnthetubaguy | bauzas: so if you can't push a patch, let me know, and I can try something, but I am just going to do some log digging on your patch | |
| 10:00:55 | bauzas | johnthetubaguy: I provided the line to look at | |
| 10:01:04 | bauzas | johnthetubaguy: in the gerrit comment | |
| 10:01:14 | johnthetubaguy | yeah, seen that | |
| 10:01:19 | bauzas | johnthetubaguy: the one matching the req-id that fails | |
| 10:01:53 | bauzas | you think it'd be better to introspect within the DB method, or outside of that? | |
| 10:02:02 | bauzas | like in the caller? | |
| 10:02:37 | bauzas | sec | |
| 10:02:40 | bauzas | I think of something | |
| 10:02:41 | johnthetubaguy | I would go for the fool proof thing, and just log out a list of services | |
| 10:02:56 | bauzas | that's a remotable method, right? | |
| 10:03:08 | bauzas | so we expect to have it run by the conductor | |
| 10:03:09 | bauzas | but | |
| 10:03:14 | bauzas | it's within the scheduler | |
| 10:03:24 | bauzas | where the indirection api is not set | |
| 10:03:32 | bauzas | so we directly lookup the DB | |
| 10:03:43 | johnthetubaguy | bauzas: is it just me, or do the old and new nodes both have the same CONF.host setting? | |
| 10:03:52 | bauzas | johnthetubaguy: it's not | |
| 10:04:05 | bauzas | johnthetubaguy: or it shouldn't | |
| 10:04:30 | bauzas | but I briefly looked once at the subnode n-cpu log, and I saw it having the right name | |
| 10:04:40 | johnthetubaguy | oops, I was on the wrong log | |
| 10:04:54 | bauzas | that test is actually pretty dummy | |
| 10:05:10 | bauzas | it just takes all the computes, and force an instance to boot against each one | |
| 10:05:22 | johnthetubaguy | http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/new/screen-n-sch.txt.gz#_2017-01-27_03_42_11_626 | |
| 10:05:28 | johnthetubaguy | http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/subnode-2/old/screen-n-cpu.txt.gz#_2017-01-27_03_31_28_948 | |
| 10:05:32 | bauzas | so it's passing on the new node, but not on the newton subnode | |
| 10:05:53 | johnthetubaguy | do we truncate the hostname? | |
| 10:06:10 | bauzas | I don't think so | |
| 10:06:32 | bauzas | oh wait | |
| 10:06:56 | bauzas | ergh, I'm almost arriving to Paris | |
| 10:07:07 | bauzas | I should loose some connectivity soon | |
| 10:07:19 | bauzas | let's add a bit of logging before my connection drops | |
| 10:13:39 | openstackgerrit | Sylvain Bauza proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | |
| 10:14:21 | bauzas | johnthetubaguy: I'm loosing connectivity, but ^ | |
| 10:14:38 | bauzas | johnthetubaguy: ttyl once I'm arriving at the office | |
| 10:15:02 | bauzas | we won't get a lot of details by that call but hopefully, we would get enough info to see if the service is found | |
| 10:15:17 | johnthetubaguy | +1 thats the hack I was thinking about | |
| 10:20:18 | openstackgerrit | Ghanshyam Mann proposed openstack/nova: Add release note and docs for filter/sort whitelist https://review.openstack.org/421760 | |
| 10:21:09 | gmann | johnthetubaguy: alex_xu mriedem : all_tenant things are ready now - https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/add-whitelist-for-server-list-filter-sort-parameters | |
| 10:21:34 | gmann | johnthetubaguy: alex_xu mriedem : updated spec and new version bump with 'no 403' on new rule | |
| 10:22:46 | gmann | johnthetubaguy: alex_xu mriedem : this one with spec patch also - https://review.openstack.org/#/q/status:open+branch:master+topic:bp/add-whitelist-for-server-list-filter-sort-parameters | |
| 10:27:38 | johnthetubaguy | gmann: thanks for fixing that up, I will try get to that | |
| 10:27:47 | johnthetubaguy | just chasing the approved FFEs | |
| 10:28:25 | gmann | johnthetubaguy: thanks, ll check tomorrow on those if any comments | |
| 10:36:27 | cdent | I'm available to be directed to whatever matters by whomever. | |
| 10:44:10 | johnthetubaguy | bauzas: http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/new/screen-n-cpu.txt#_2017-01-27_03_42_12_029 | |
| 10:44:51 | johnthetubaguy | cdent: I think the priorities are the custom resource patches and bauzas's call placement patch | |
| 10:45:10 | johnthetubaguy | seems like we get the min_service version wrong somehow | |
| 10:45:13 | johnthetubaguy | but the plot thickens | |
| 10:45:29 | cdent | eeeph | |
| 10:45:35 | johnthetubaguy | bauzas: then later you see this: http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/new/screen-n-cpu.txt#_2017-01-27_03_42_14_659 | |
| 10:48:23 | johnthetubaguy | I mean if the min_service version stuff is broken, thats a big deal for other things | |
| 10:48:35 | johnthetubaguy | so it could be a good find, or a total red herring, but thats life I guess | |
| 10:55:47 | openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | |
| 10:56:35 | openstackgerrit | Ildiko Vancsa proposed openstack/nova: Implement new attach/detach Cinder flow https://review.openstack.org/330285 | |
| 10:56:38 | openstackgerrit | Ildiko Vancsa proposed openstack/nova: Remove check_attach https://review.openstack.org/335358 | |
| 10:59:23 | openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | |
| 11:12:00 | cdent | johnthetubaguy: It looks like edleafe has addressed mriedem's concers on https://review.openstack.org/#/c/404472/ and I just gave it read through and nothing lept out as crazy, and the voiting bits of jenkins are happy, so it might be gtg | |
| 11:12:21 | johnthetubaguy | cdent: ah, sweet, I should look into that one | |
| 11:18:06 | johnthetubaguy | cdent: edleafe: I wonder if we should add a reno note for that patch, not a blocker as it can be a follow on, but worth squaring that away | |
| 11:20:20 | cdent | johnthetubaguy: perhaps, but if we're looking for an excuse to why not, this change has no visible change on end behavior (you have to look in the db or placement api with specific intent to know). | |
| 11:21:39 | johnthetubaguy | cdent: good point | |
| 11:21:56 | johnthetubaguy | "hey that thing you didn't know way rubbish, we fixed it, yey us" | |
| 11:22:14 | cdent | heh | |
| 11:24:01 | cdent | bauzas, johnthetubaguy : the current strategy on the scheduler changes is to watch the debug output and from that figure out what's next? | |
| 11:24:36 | johnthetubaguy | cdent: yeah | |
| 11:25:02 | johnthetubaguy | I tralled through the code and logs a bit this morning, haven't seem any more clues as such, just more things that confuse me | |
| 11:25:29 | johnthetubaguy | I haven't double checked that we correctly have the min_version caching working, that was one thread I haven't double checked | |
| 11:27:11 | cdent | is here a chance that the old version can be cached, leaving the cache wrong? that is, the scheduler starts up while a node is old and caches the bad version? | |
| 11:27:18 | cdent | we could test that by turning off the caching? | |
| 11:30:16 | cdent | that does seem possible | |
| 11:30:25 | cdent | assuming the cache is local to the current process | |
| 11:34:02 | cdent | johnthetubaguy: yeah, I'm pretty sure the above is possible, because unlike the conductor, the scheduler isn't required to listen to a hup to reset the cache. This is the review that put the cache in https://review.openstack.org/#/c/265326/ | |
| 11:35:06 | johnthetubaguy | cdent: so in this case we want to ensure we get the older version | |
| 11:35:15 | johnthetubaguy | cdent: so I was expecting the scheduler to help really | |
| 11:35:24 | johnthetubaguy | I mean the cache to help | |