Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-23
15:08:16 mriedem but it comes up as a problem about every 3 months :)
15:08:23 Kevin_Zheng ：）
15:08:37 Kevin_Zheng but still
15:08:58 Kevin_Zheng we will have none in cinder if not provide one
15:09:01 bauzas gibi: yup, I reviewed it even :)
15:09:17 Kevin_Zheng anyway to sync it？
15:09:23 gibi bauzas: I think that spec contains a lot of nice idea and discussion on this matter
15:11:45 mriedem Kevin_Zheng: let me check the cinder api
15:11:58 mriedem if mountpoint is required then we'd have an issue
15:12:52 Kevin_Zheng mriedem: Im using phone so I can't send you the code，but there is a if condation in
15:13:14 Kevin_Zheng nova's attach function
15:13:59 Kevin_Zheng it checked volume.attach_status =detached
15:14:36 Kevin_Zheng maybe we can also check device name there，if none，send again the
15:14:42 Kevin_Zheng right one
15:15:48 openstackgerrit Maciej Szankin proposed openstack/nova-specs master: Add deprecate-nova-cert spec https://review.openstack.org/448882
15:16:49 openstackgerrit Maciej Szankin proposed openstack/nova-specs master: Add deprecate-nova-cert spec https://review.openstack.org/448882
15:19:02 mriedem Kevin_Zheng: looks like the mountpoint is required in the os-attach API in cinder
15:19:23 mriedem in fact, it looks like i can trigger a 500 error if i don't include it in the request body
15:26:13 mriedem melwitt: dansmith: I think Kevin_Zheng found another fun local delete cell0 issue :)
15:26:44 mriedem not totally sure yet, but we don't call http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2144 when doing the local delete case where the instance isn't in a cell
15:26:56 mriedem so the instance is in error state but has volumes attached, and we don't cleanup the volumes
15:27:19 mriedem i think this would be (1) create server, (2) shelve it, (3) attach volume (4) unshelve - fails, puts instance in error state (5) delete
15:27:40 mriedem the question is, does the instance got to cell0 on the unshelve failure in step 4?
15:28:24 dansmith shouldn't,
15:28:51 dansmith because you can't move the instance, so it wouldn't reschedule to cell0 if it's already in cell1, which it has to be because it was shelved in the first place
15:29:05 mriedem madhaviy: yes the policy check happens on the compute, which is odd, i've brought this up in the mailing list before, i'd have to dig that up
15:29:20 mriedem dansmith: but on shelve offload the instance no longer has a host right?
15:29:25 mriedem althought,
15:29:28 mriedem it should have an instance mapping
15:29:31 mriedem back to the cell it's in
15:29:32 dansmith mriedem: sure, but it's in a cell
15:29:39 mriedem ok
15:29:41 dansmith you can't move the instance record from one db to the other Imeant
15:29:59 mriedem but a shelved offloaded instance will have an instance mapping,
15:30:01 mriedem so it's in a cell
15:30:04 mriedem even if instance.host is NOne
15:30:10 madhaviy mriedem: sure
15:30:18 dansmith right
15:31:05 mriedem so in that case,
15:31:09 mriedem on delete, we still get here http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n1847
15:31:19 mriedem and the cell could be cell1 or whatever
15:31:39 mriedem and we just destroy the instance here http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n1887
15:32:37 mriedem which bypasses http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2206
15:32:48 mriedem if i'm following this correctly
15:33:09 mriedem basically, for local delete we've duplicated the paths, and are missing some things from the case that we don't have instance.host but it's in a cell
15:34:08 mriedem i think i could write a functional test for this pretty easily,
15:34:20 mriedem and verify if the bdm was destroyed when the instance was deleted or not
16:07:38 mriedem weird, the python-novaclient jobs are timing out since 3/21
16:07:49 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Killed%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20timeout%20-s%209%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20project%3A%5C%22openstack%2Fpython-novaclient%5C%22&from=7d
16:10:13 openstackgerrit Maciej Szankin proposed openstack/nova-specs master: Add remove-nova-cert spec https://review.openstack.org/448882
16:10:54 openstackgerrit Maciej Szankin proposed openstack/nova-specs master: Add remove-nova-cert spec https://review.openstack.org/448882
16:11:59 sdague mriedem: well there are some crash dump tests in there that are taking 4 minutes to run
16:12:52 sdague mriedem: http://logs.openstack.org/41/449041/1/check/gate-novaclient-dsvm-functional-identity-v3-only-ubuntu-xenial-nv/191f2b4/console.html#_2017-03-23_11_04_17_455195
16:13:07 sdague something else is going on, the tests passed, then zuul forgot about it for an hour
16:14:06 mriedem yeah
16:14:07 mriedem 2017-03-23 11:04:17.455195 | 2017-03-23 11:04:17.449 | Finished
16:14:15 mriedem 2017-03-23 12:00:48.459991 | /home/jenkins/workspace/gate-novaclient-dsvm-functional-identity-v3-only-ubuntu-xenial-nv/devstack-gate/functions.sh: line 1074: 3542 Killed
16:14:22 mriedem so 56 minutes of nothing
16:14:34 sdague I saw a thing like that before and brought it up in infra, I can't remember now what was getting lost
16:21:40 efried harlowja One thing I'm finding with TaskFlow is it's kinda tricky to unit test around.
16:22:07 efried Especially with nested flows and such.
16:23:22 efried If I mock the whole TaskFlow apparatus, I have to test the individual tasks separately and can't get good verification of the overall flow with dependencies and such.
16:23:36 johnthetubaguy tojuvone: are you still about, I forgot to ping you when I got back
16:24:00 tojuvone hieulq, yes, still here
16:24:39 edmondsw johnthetubaguy are you aware that there is a policy rule (network:attach_external_network) being checked by nova-compute?
16:24:42 efried harlowja But if I don't, I better not try to mock any of the individual tasks - too much internal stuff is expected.
16:25:16 efried Not a show-stopper. And not sure what could be done to mitigate - maybe taskflow lib providing some stock fixtures?
16:25:20 sdague mriedem: are there no individual test timeouts on these?
16:25:35 edmondsw johnthetubaguy just found that doesn't work if I don't put a policy file on a host... ugh... we should only be checking policy in nova-api
16:25:43 johnthetubaguy edmondsw: you reminded me, I forgot about that one again, there are a few strange stragglers like that
16:25:53 tojuvone johnthetubaguy, yes, so was wondering the maintenance_events. Wither own table or in services table
16:26:17 tojuvone johnthetubaguy, services exposed to servers so that works
16:26:27 mriedem sdague: i'm not seeing any
16:26:30 edmondsw johnthetubaguy that's gotta be a defect, right? There's even talk about nova-compute being split out from nova. It shouldn't be relying on nova's policy file
16:26:37 tojuvone johnthetubaguy, but in services need to duplicate on each service on host
16:26:39 mriedem sdague: i'm seeing rabbit have memory limit warnings around the time we lose the logs
16:27:19 sdague yeh, there is no individual test timeout, the test is still running and hung
16:27:19 mriedem edmondsw: see my reply to madhaviy above
16:27:36 edmondsw mriedem, ah, didn't realize we'd already had this discussion...
16:27:37 mriedem sdague: doesn't testr have a 500s timeout per test?
16:27:45 johnthetubaguy tojuvone: given what I described, I think it would get its own table, but not sure if everyone likes that direction, I think we should focus on getting the use cases crisp first
16:28:31 mriedem edmondsw: madhaviy: johnthetubaguy: https://openstack.nimeyo.com/87011/openstack-policy-check-network-attach_external_network
16:28:31 openstackgerrit Sean Dague proposed openstack/python-novaclient master: set test timout to 300 seconds https://review.openstack.org/449211
16:28:39 sdague mriedem: no, testr enforces nothing
16:28:41 tojuvone johnthetubaguy, yes, trying to update a lot including the cases and all.
16:29:26 johnthetubaguy tojuvone: yeah, I would just leave the implementation bit alone for now, lets agree these use cases first, particularly the scheduler related ones
16:29:34 sdague mriedem: all our test timeouts are done via test fixtures
16:29:39 mriedem sdague: ah
16:29:45 mriedem i see we set that for nova but not novaclient
16:29:48 sdague yep
16:30:01 johnthetubaguy edmondsw: there were other ones I think sneti and aunnam found that are accessed outside the API
16:30:01 mriedem self.useFixture(fixtures.Timeout(test_timeout, gentle=True))
16:30:01 mriedem yar
16:30:09 openstackgerrit Dan Smith proposed openstack/nova master: Get instance availability_zone without hitting the api db https://review.openstack.org/439754
16:30:09 openstackgerrit Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162
16:30:10 openstackgerrit Dan Smith proposed openstack/nova master: Set instance.availability_zone whenever we schedule https://review.openstack.org/446053
16:30:11 openstackgerrit Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891
16:30:12 tojuvone johnthetubaguy, ok, sounds good to make better progress
16:30:23 openstackgerrit Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174
16:30:25 openstackgerrit Dan Smith proposed openstack/nova master: Add get_by_instance_uuids() to InstanceMappingList https://review.openstack.org/443292

Earlier   Later