| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-23 | |||
| 15:06:30 | gibi | bauzas: there was a proposal from jaypipes to revisit the server group concept here https://review.openstack.org/#/c/183837/4/specs/liberty/approved/generic-scheduling-policies.rst | |
| 15:06:37 | mriedem | Kevin_Zheng: i think that might still be a lie though, | |
| 15:06:52 | mriedem | because just because the user asked for /dev/vda doesn't mean that's where it will show up on the guest | |
| 15:06:58 | mriedem | even though we stored that in cinder | |
| 15:07:03 | Kevin_Zheng | yeah | |
| 15:07:16 | mriedem | this is really why we need to just have a microversion to take device_name out of the volume attach API :) | |
| 15:07:21 | mriedem | because we can't honor it | |
| 15:07:27 | mriedem | we don't even try with the libvirt driver | |
| 15:07:29 | mriedem | we ignore it | |
| 15:07:32 | Kevin_Zheng | yeah | |
| 15:07:45 | Kevin_Zheng | I will try to put up one | |
| 15:07:56 | Kevin_Zheng | if nobody does | |
| 15:08:03 | mriedem | thanks, this is something we've talked about in previous summits and meetups and just never had someone write a spec | |
| 15:08:16 | mriedem | but it comes up as a problem about every 3 months :) | |
| 15:08:23 | Kevin_Zheng | ：） | |
| 15:08:37 | Kevin_Zheng | but still | |
| 15:08:58 | Kevin_Zheng | we will have none in cinder if not provide one | |
| 15:09:01 | bauzas | gibi: yup, I reviewed it even :) | |
| 15:09:17 | Kevin_Zheng | anyway to sync it？ | |
| 15:09:23 | gibi | bauzas: I think that spec contains a lot of nice idea and discussion on this matter | |
| 15:11:45 | mriedem | Kevin_Zheng: let me check the cinder api | |
| 15:11:58 | mriedem | if mountpoint is required then we'd have an issue | |
| 15:12:52 | Kevin_Zheng | mriedem: Im using phone so I can't send you the code，but there is a if condation in | |
| 15:13:14 | Kevin_Zheng | nova's attach function | |
| 15:13:59 | Kevin_Zheng | it checked volume.attach_status =detached | |
| 15:14:36 | Kevin_Zheng | maybe we can also check device name there，if none，send again the | |
| 15:14:42 | Kevin_Zheng | right one | |
| 15:15:48 | openstackgerrit | Maciej Szankin proposed openstack/nova-specs master: Add deprecate-nova-cert spec https://review.openstack.org/448882 | |
| 15:16:49 | openstackgerrit | Maciej Szankin proposed openstack/nova-specs master: Add deprecate-nova-cert spec https://review.openstack.org/448882 | |
| 15:19:02 | mriedem | Kevin_Zheng: looks like the mountpoint is required in the os-attach API in cinder | |
| 15:19:23 | mriedem | in fact, it looks like i can trigger a 500 error if i don't include it in the request body | |
| 15:26:13 | mriedem | melwitt: dansmith: I think Kevin_Zheng found another fun local delete cell0 issue :) | |
| 15:26:44 | mriedem | not totally sure yet, but we don't call http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2144 when doing the local delete case where the instance isn't in a cell | |
| 15:26:56 | mriedem | so the instance is in error state but has volumes attached, and we don't cleanup the volumes | |
| 15:27:19 | mriedem | i think this would be (1) create server, (2) shelve it, (3) attach volume (4) unshelve - fails, puts instance in error state (5) delete | |
| 15:27:40 | mriedem | the question is, does the instance got to cell0 on the unshelve failure in step 4? | |
| 15:28:24 | dansmith | shouldn't, | |
| 15:28:51 | dansmith | because you can't move the instance, so it wouldn't reschedule to cell0 if it's already in cell1, which it has to be because it was shelved in the first place | |
| 15:29:05 | mriedem | madhaviy: yes the policy check happens on the compute, which is odd, i've brought this up in the mailing list before, i'd have to dig that up | |
| 15:29:20 | mriedem | dansmith: but on shelve offload the instance no longer has a host right? | |
| 15:29:25 | mriedem | althought, | |
| 15:29:28 | mriedem | it should have an instance mapping | |
| 15:29:31 | mriedem | back to the cell it's in | |
| 15:29:32 | dansmith | mriedem: sure, but it's in a cell | |
| 15:29:39 | mriedem | ok | |
| 15:29:41 | dansmith | you can't move the instance record from one db to the other Imeant | |
| 15:29:59 | mriedem | but a shelved offloaded instance will have an instance mapping, | |
| 15:30:01 | mriedem | so it's in a cell | |
| 15:30:04 | mriedem | even if instance.host is NOne | |
| 15:30:10 | madhaviy | mriedem: sure | |
| 15:30:18 | dansmith | right | |
| 15:31:05 | mriedem | so in that case, | |
| 15:31:09 | mriedem | on delete, we still get here http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n1847 | |
| 15:31:19 | mriedem | and the cell could be cell1 or whatever | |
| 15:31:39 | mriedem | and we just destroy the instance here http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n1887 | |
| 15:32:37 | mriedem | which bypasses http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2206 | |
| 15:32:48 | mriedem | if i'm following this correctly | |
| 15:33:09 | mriedem | basically, for local delete we've duplicated the paths, and are missing some things from the case that we don't have instance.host but it's in a cell | |
| 15:34:08 | mriedem | i think i could write a functional test for this pretty easily, | |
| 15:34:20 | mriedem | and verify if the bdm was destroyed when the instance was deleted or not | |
| 16:07:38 | mriedem | weird, the python-novaclient jobs are timing out since 3/21 | |
| 16:07:49 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Killed%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20timeout%20-s%209%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fpython-novaclient%5C%22&from=7d | |
| 16:10:13 | openstackgerrit | Maciej Szankin proposed openstack/nova-specs master: Add remove-nova-cert spec https://review.openstack.org/448882 | |
| 16:10:54 | openstackgerrit | Maciej Szankin proposed openstack/nova-specs master: Add remove-nova-cert spec https://review.openstack.org/448882 | |
| 16:11:59 | sdague | mriedem: well there are some crash dump tests in there that are taking 4 minutes to run | |
| 16:12:52 | sdague | mriedem: http://logs.openstack.org/41/449041/1/check/gate-novaclient-dsvm-functional-identity-v3-only-ubuntu-xenial-nv/191f2b4/console.html#_2017-03-23_11_04_17_455195 | |
| 16:13:07 | sdague | something else is going on, the tests passed, then zuul forgot about it for an hour | |
| 16:14:06 | mriedem | yeah | |
| 16:14:07 | mriedem | 2017-03-23 11:04:17.455195 | 2017-03-23 11:04:17.449 | Finished | |
| 16:14:15 | mriedem | 2017-03-23 12:00:48.459991 | /home/jenkins/workspace/gate-novaclient-dsvm-functional-identity-v3-only-ubuntu-xenial-nv/devstack-gate/functions.sh: line 1074: 3542 Killed | |
| 16:14:22 | mriedem | so 56 minutes of nothing | |
| 16:14:34 | sdague | I saw a thing like that before and brought it up in infra, I can't remember now what was getting lost | |
| 16:21:40 | efried | harlowja One thing I'm finding with TaskFlow is it's kinda tricky to unit test around. | |
| 16:22:07 | efried | Especially with nested flows and such. | |
| 16:23:22 | efried | If I mock the whole TaskFlow apparatus, I have to test the individual tasks separately and can't get good verification of the overall flow with dependencies and such. | |
| 16:23:36 | johnthetubaguy | tojuvone: are you still about, I forgot to ping you when I got back | |
| 16:24:00 | tojuvone | hieulq, yes, still here | |
| 16:24:39 | edmondsw | johnthetubaguy are you aware that there is a policy rule (network:attach_external_network) being checked by nova-compute? | |
| 16:24:42 | efried | harlowja But if I don't, I better not try to mock any of the individual tasks - too much internal stuff is expected. | |
| 16:25:16 | efried | Not a show-stopper. And not sure what could be done to mitigate - maybe taskflow lib providing some stock fixtures? | |
| 16:25:20 | sdague | mriedem: are there no individual test timeouts on these? | |
| 16:25:35 | edmondsw | johnthetubaguy just found that doesn't work if I don't put a policy file on a host... ugh... we should only be checking policy in nova-api | |
| 16:25:43 | johnthetubaguy | edmondsw: you reminded me, I forgot about that one again, there are a few strange stragglers like that | |
| 16:25:53 | tojuvone | johnthetubaguy, yes, so was wondering the maintenance_events. Wither own table or in services table | |
| 16:26:17 | tojuvone | johnthetubaguy, services exposed to servers so that works | |
| 16:26:27 | mriedem | sdague: i'm not seeing any | |
| 16:26:30 | edmondsw | johnthetubaguy that's gotta be a defect, right? There's even talk about nova-compute being split out from nova. It shouldn't be relying on nova's policy file | |
| 16:26:37 | tojuvone | johnthetubaguy, but in services need to duplicate on each service on host | |
| 16:26:39 | mriedem | sdague: i'm seeing rabbit have memory limit warnings around the time we lose the logs | |
| 16:27:19 | sdague | yeh, there is no individual test timeout, the test is still running and hung | |
| 16:27:19 | mriedem | edmondsw: see my reply to madhaviy above | |
| 16:27:36 | edmondsw | mriedem, ah, didn't realize we'd already had this discussion... | |
| 16:27:37 | mriedem | sdague: doesn't testr have a 500s timeout per test? | |
| 16:27:45 | johnthetubaguy | tojuvone: given what I described, I think it would get its own table, but not sure if everyone likes that direction, I think we should focus on getting the use cases crisp first | |
| 16:28:31 | mriedem | edmondsw: madhaviy: johnthetubaguy: https://openstack.nimeyo.com/87011/openstack-policy-check-network-attach_external_network | |
| 16:28:31 | openstackgerrit | Sean Dague proposed openstack/python-novaclient master: set test timout to 300 seconds https://review.openstack.org/449211 | |
| 16:28:39 | sdague | mriedem: no, testr enforces nothing | |
| 16:28:41 | tojuvone | johnthetubaguy, yes, trying to update a lot including the cases and all. | |
| 16:29:26 | johnthetubaguy | tojuvone: yeah, I would just leave the implementation bit alone for now, lets agree these use cases first, particularly the scheduler related ones | |
| 16:29:34 | sdague | mriedem: all our test timeouts are done via test fixtures | |