| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-23 | |||
| 15:29:59 | mriedem | but a shelved offloaded instance will have an instance mapping, | |
| 15:30:01 | mriedem | so it's in a cell | |
| 15:30:04 | mriedem | even if instance.host is NOne | |
| 15:30:10 | madhaviy | mriedem: sure | |
| 15:30:18 | dansmith | right | |
| 15:31:05 | mriedem | so in that case, | |
| 15:31:09 | mriedem | on delete, we still get here http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n1847 | |
| 15:31:19 | mriedem | and the cell could be cell1 or whatever | |
| 15:31:39 | mriedem | and we just destroy the instance here http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n1887 | |
| 15:32:37 | mriedem | which bypasses http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2206 | |
| 15:32:48 | mriedem | if i'm following this correctly | |
| 15:33:09 | mriedem | basically, for local delete we've duplicated the paths, and are missing some things from the case that we don't have instance.host but it's in a cell | |
| 15:34:08 | mriedem | i think i could write a functional test for this pretty easily, | |
| 15:34:20 | mriedem | and verify if the bdm was destroyed when the instance was deleted or not | |
| 16:07:38 | mriedem | weird, the python-novaclient jobs are timing out since 3/21 | |
| 16:07:49 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Killed%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20timeout%20-s%209%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fpython-novaclient%5C%22&from=7d | |
| 16:10:13 | openstackgerrit | Maciej Szankin proposed openstack/nova-specs master: Add remove-nova-cert spec https://review.openstack.org/448882 | |
| 16:10:54 | openstackgerrit | Maciej Szankin proposed openstack/nova-specs master: Add remove-nova-cert spec https://review.openstack.org/448882 | |
| 16:11:59 | sdague | mriedem: well there are some crash dump tests in there that are taking 4 minutes to run | |
| 16:12:52 | sdague | mriedem: http://logs.openstack.org/41/449041/1/check/gate-novaclient-dsvm-functional-identity-v3-only-ubuntu-xenial-nv/191f2b4/console.html#_2017-03-23_11_04_17_455195 | |
| 16:13:07 | sdague | something else is going on, the tests passed, then zuul forgot about it for an hour | |
| 16:14:06 | mriedem | yeah | |
| 16:14:07 | mriedem | 2017-03-23 11:04:17.455195 | 2017-03-23 11:04:17.449 | Finished | |
| 16:14:15 | mriedem | 2017-03-23 12:00:48.459991 | /home/jenkins/workspace/gate-novaclient-dsvm-functional-identity-v3-only-ubuntu-xenial-nv/devstack-gate/functions.sh: line 1074: 3542 Killed | |
| 16:14:22 | mriedem | so 56 minutes of nothing | |
| 16:14:34 | sdague | I saw a thing like that before and brought it up in infra, I can't remember now what was getting lost | |
| 16:21:40 | efried | harlowja One thing I'm finding with TaskFlow is it's kinda tricky to unit test around. | |
| 16:22:07 | efried | Especially with nested flows and such. | |
| 16:23:22 | efried | If I mock the whole TaskFlow apparatus, I have to test the individual tasks separately and can't get good verification of the overall flow with dependencies and such. | |
| 16:23:36 | johnthetubaguy | tojuvone: are you still about, I forgot to ping you when I got back | |
| 16:24:00 | tojuvone | hieulq, yes, still here | |
| 16:24:39 | edmondsw | johnthetubaguy are you aware that there is a policy rule (network:attach_external_network) being checked by nova-compute? | |
| 16:24:42 | efried | harlowja But if I don't, I better not try to mock any of the individual tasks - too much internal stuff is expected. | |
| 16:25:16 | efried | Not a show-stopper. And not sure what could be done to mitigate - maybe taskflow lib providing some stock fixtures? | |
| 16:25:20 | sdague | mriedem: are there no individual test timeouts on these? | |
| 16:25:35 | edmondsw | johnthetubaguy just found that doesn't work if I don't put a policy file on a host... ugh... we should only be checking policy in nova-api | |
| 16:25:43 | johnthetubaguy | edmondsw: you reminded me, I forgot about that one again, there are a few strange stragglers like that | |
| 16:25:53 | tojuvone | johnthetubaguy, yes, so was wondering the maintenance_events. Wither own table or in services table | |
| 16:26:17 | tojuvone | johnthetubaguy, services exposed to servers so that works | |
| 16:26:27 | mriedem | sdague: i'm not seeing any | |
| 16:26:30 | edmondsw | johnthetubaguy that's gotta be a defect, right? There's even talk about nova-compute being split out from nova. It shouldn't be relying on nova's policy file | |
| 16:26:37 | tojuvone | johnthetubaguy, but in services need to duplicate on each service on host | |
| 16:26:39 | mriedem | sdague: i'm seeing rabbit have memory limit warnings around the time we lose the logs | |
| 16:27:19 | mriedem | edmondsw: see my reply to madhaviy above | |
| 16:27:19 | sdague | yeh, there is no individual test timeout, the test is still running and hung | |
| 16:27:36 | edmondsw | mriedem, ah, didn't realize we'd already had this discussion... | |
| 16:27:37 | mriedem | sdague: doesn't testr have a 500s timeout per test? | |
| 16:27:45 | johnthetubaguy | tojuvone: given what I described, I think it would get its own table, but not sure if everyone likes that direction, I think we should focus on getting the use cases crisp first | |
| 16:28:31 | openstackgerrit | Sean Dague proposed openstack/python-novaclient master: set test timout to 300 seconds https://review.openstack.org/449211 | |
| 16:28:31 | mriedem | edmondsw: madhaviy: johnthetubaguy: https://openstack.nimeyo.com/87011/openstack-policy-check-network-attach_external_network | |
| 16:28:39 | sdague | mriedem: no, testr enforces nothing | |
| 16:28:41 | tojuvone | johnthetubaguy, yes, trying to update a lot including the cases and all. | |
| 16:29:26 | johnthetubaguy | tojuvone: yeah, I would just leave the implementation bit alone for now, lets agree these use cases first, particularly the scheduler related ones | |
| 16:29:34 | sdague | mriedem: all our test timeouts are done via test fixtures | |
| 16:29:39 | mriedem | sdague: ah | |
| 16:29:45 | mriedem | i see we set that for nova but not novaclient | |
| 16:29:48 | sdague | yep | |
| 16:30:01 | mriedem | yar | |
| 16:30:01 | mriedem | self.useFixture(fixtures.Timeout(test_timeout, gentle=True)) | |
| 16:30:01 | johnthetubaguy | edmondsw: there were other ones I think sneti and aunnam found that are accessed outside the API | |
| 16:30:09 | openstackgerrit | Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162 | |
| 16:30:09 | openstackgerrit | Dan Smith proposed openstack/nova master: Get instance availability_zone without hitting the api db https://review.openstack.org/439754 | |
| 16:30:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Set instance.availability_zone whenever we schedule https://review.openstack.org/446053 | |
| 16:30:11 | openstackgerrit | Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891 | |
| 16:30:12 | tojuvone | johnthetubaguy, ok, sounds good to make better progress | |
| 16:30:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174 | |
| 16:30:25 | openstackgerrit | Dan Smith proposed openstack/nova master: Add get_by_instance_uuids() to InstanceMappingList https://review.openstack.org/443292 | |
| 16:30:26 | openstackgerrit | Dan Smith proposed openstack/nova master: Clean up ClientRouter debt https://review.openstack.org/444487 | |
| 16:30:27 | openstackgerrit | Dan Smith proposed openstack/nova master: Make server_groups determine deleted-ness from InstanceMappingList https://review.openstack.org/443293 | |
| 16:30:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Remove Mitaka-era service version check https://review.openstack.org/442861 | |
| 16:30:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Make conductor ask scheduler to limit migrates to same cell https://review.openstack.org/438025 | |
| 16:30:31 | openstackgerrit | Dan Smith proposed openstack/nova master: Add workaround to disable group policy check upcall https://review.openstack.org/442736 | |
| 16:30:44 | sdague | mriedem: that should at least give us different results, and not have the big timeout kill us | |
| 16:30:56 | mriedem | sdague: yeah, thanks | |
| 16:31:02 | johnthetubaguy | edmondsw: FWIW, I think the neutron refactor work is heading in a direction that would fix that policy issue, anyways | |
| 16:31:14 | sneti | johnthetubaguy, yes..this is the one: https://review.openstack.org/#/c/448260/2/nova/policies/cells_scheduler.py | |
| 16:31:40 | sdague | mriedem: is the crash dump testing a thing we do other places? | |
| 16:31:54 | tojuvone | johnthetubaguy, I should have that part by tomorrow and can continue from that | |
| 16:31:54 | mriedem | sdague: not sure about tempest | |
| 16:32:06 | johnthetubaguy | tojuvone: sounds great, ping me when thats up | |
| 16:32:11 | mriedem | sdague: oomichi probably knows, i think he added it | |
| 16:32:18 | mriedem | or was involved | |
| 16:32:28 | tojuvone | johnthetubaguy, thanks, I will | |
| 16:33:03 | johnthetubaguy | edmondsw: so sounds like some scheduler filters need policy too | |
| 16:33:38 | johnthetubaguy | edmondsw: that should probably get fixed too | |
| 16:34:16 | edmondsw | johnthetubaguy I'd say let's open defects for these, so they're not lost... and if they get fixed apart from the defects we can always cancel them. Your thoughts? | |
| 16:34:43 | johnthetubaguy | edmondsw: yeah, I have asked sneti and aunnam to do that as they find these | |
| 16:34:51 | edmondsw | great, tx | |
| 16:35:03 | johnthetubaguy | or at least I mean to... | |
| 16:35:13 | edmondsw | sneti, have you opened one for the external network attach case, or will you, or should I? | |
| 16:35:48 | edmondsw | sneti, if you haven't already, I'd say you open the scheduler one you found and I can open the network one | |
| 16:36:05 | oomichi | mriedem: sdague: temepst doesn't have crash-dump test, I guess cirros cannot take crash-dump because it requires some crash-dump tool | |
| 16:36:11 | johnthetubaguy | edmondsw: +1 | |
| 16:36:19 | oomichi | even if nova triggers it | |
| 16:36:50 | sdague | oomichi: ok, so why are we running those tests? | |
| 16:36:51 | mriedem | oomichi: but we have crash dump tests in novaclient functional tests | |
| 16:37:38 | oomichi | mriedem: oh, I didn't that novaclient has | |
| 16:39:15 | oomichi | sdague: that depends on what kind of test do we want. Cirros is enough if we just want to call the API I feel. but if we want to verify the crashdump file is created in the virtual machine, the tool is required | |
| 16:39:19 | mriedem | oomichi: yeah and they take about 4 minutes to run | |
| 16:39:35 | mriedem | http://logs.openstack.org/57/447657/1/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/b3aaf58/console.html#_2017-03-23_12_21_00_379603 | |