| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-14 | |||
| 17:22:53 | mriedem | not in minnesota | |
| 17:23:21 | bauzas | Minnesota is that better from other states that it saves 8 mins for every worker ? | |
| 17:23:29 | stephenfin | bauzas: But yeah, that call ends up here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1436 | |
| 17:23:36 | mriedem | bauzas: it is; that's why our taxes are high | |
| 17:24:02 | bauzas | mriedem: I'm sure we beat your state | |
| 17:24:04 | stephenfin | which then calls in here, where we _eventually_ raise the exception https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1513 | |
| 17:24:20 | mriedem | bauzas: sure, but, | |
| 17:24:31 | mriedem | we don't get 50% of the month off for PTO :P | |
| 17:25:04 | bauzas | jeez | |
| 17:25:07 | bauzas | touché | |
| 17:25:53 | bauzas | stephenfin: so, give me a sec to rephrase that | |
| 17:26:12 | bauzas | stephenfin: you mean that the policy check will be done primarly by the scheduler then ? | |
| 17:26:31 | bauzas | I need to rereview then the implications | |
| 17:26:35 | stephenfin | It'll be done twice - once by the scheduler and once when we're booting | |
| 17:26:45 | stephenfin | Because races are possible | |
| 17:26:49 | bauzas | yeah, but totally not at the same time :) | |
| 17:26:57 | stephenfin | Indeed :) | |
| 17:27:18 | stephenfin | So, per above, what a user will see is a failure to find suitable hosts | |
| 17:27:19 | bauzas | so, say we raise an exception, all of that stuff if called by the filter, right? | |
| 17:27:35 | stephenfin | All of what stuff? | |
| 17:27:35 | bauzas | s/if/is | |
| 17:28:37 | bauzas | stephenfin: https://review.openstack.org/#/c/527472/4/nova/virt/hardware.py@1197 is called eventually by the filter ? | |
| 17:28:56 | stephenfin | Yes | |
| 17:29:09 | stephenfin | Ditto for https://review.openstack.org/#/c/527472/4/nova/virt/hardware.py@1184 | |
| 17:29:44 | bauzas | stephenfin: so that worst than a reschedule | |
| 17:29:49 | bauzas | that's | |
| 17:30:05 | bauzas | stephenfin: if the filter raises an exception, the whole scheduling process fails | |
| 17:30:34 | bauzas | I need to test that, but I'm pretty it goes into ERROR/scheduling | |
| 17:30:46 | bauzas | oh man, fat fingers | |
| 17:31:43 | vivsoni__ | mriedem: Hi | |
| 17:32:37 | stephenfin | bauzas: Ack. I'm going to test it now myself with CPU policies | |
| 17:32:51 | bauzas | stephenfin: I need to bail out, family duty | |
| 17:32:56 | bauzas | ++ | |
| 17:33:10 | stephenfin | A-ok. I'm gone tomorrow but I'll be around Monday if you've questions | |
| 17:33:31 | mriedem | vivsoni__: hi | |
| 17:34:04 | bauzas | stephenfin: drop me a note about your findings and if that's not a full scheduling break, then +2 on the last PS | |
| 17:34:20 | stephenfin | bauzas: Spot on. Enjoy family time :) | |
| 17:34:28 | bauzas | if that's a full scheduling stop then I'm pretty -1 on the existing | |
| 17:34:47 | bauzas | or make the filter resilient to errors | |
| 17:35:04 | vivsoni__ | mriedem: Need your input in resolving tempest Test case "tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON.test_detach_volume_shelved_or_offload_server" | |
| 17:35:25 | vivsoni__ | hpe ci is failing because of that | |
| 17:37:05 | openstackgerrit | Ade Lee proposed openstack/nova master: Add flag to make glance image verification optional https://review.openstack.org/528017 | |
| 17:39:32 | vivsoni__ | mriedem: In this TC 'connector', which is passed to cinder vendor driver's 'initialize_connection' & 'terminate_connection' is empty | |
| 17:43:21 | vivsoni__ | basically the driver's 'def initialize_connection' is not called, when 'attach_volume' is called https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L252 | |
| 17:43:59 | vivsoni__ | its failing here https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L253 | |
| 17:46:01 | mriedem | vivsoni__: that's because nova doesn't call initialize_connection anymore | |
| 17:46:04 | mriedem | it calls attachment_update | |
| 17:47:02 | vivsoni__ | this detach_volume(), internally calls to 'def terminate_connection()' of cinder driver's and failed when tries to get connector['host'] | |
| 17:48:25 | mriedem | vivsoni__: this would be easier if you could point me at CI job logs | |
| 17:48:27 | mriedem | with an actual failure | |
| 17:48:41 | mriedem | i am woefully behind on my todo list for today | |
| 17:49:07 | openstackgerrit | Merged openstack/nova stable/ocata: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/525666 | |
| 17:50:12 | vivsoni__ | mriedem: request you to please have a look at http://54.201.44.218/38/527338/2/check/3par-fc-driver-master-client-pip-c8k01-dsvm/c156eb1/ | |
| 17:53:18 | mriedem | vivsoni__: ok so it's attaching a volume to a shelved offloaded server, | |
| 17:53:24 | mriedem | so there is no host connector because there is no compute host for that server, | |
| 17:53:35 | mriedem | and then the test detaches the volume while the server is still shelved offloaded | |
| 17:53:44 | mriedem | and that's where it breaks right? | |
| 17:53:59 | vivsoni__ | mriedem: yes | |
| 17:54:09 | mriedem | ok that's right here v | |
| 17:54:10 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1994 | |
| 17:54:19 | mriedem | that's the new call to cinder in this case | |
| 17:54:31 | vivsoni__ | basically volume is not attach and tries to detaches and failed | |
| 17:54:56 | openstackgerrit | Merged openstack/nova stable/newton: Add regression test for rebuilding a volume-backed server https://review.openstack.org/526542 | |
| 17:55:09 | mriedem | vivsoni__: i'm not sure why this would be any different from a nova perspective - when attaching a volume to a shelved offloaded server, not never calls os-initialize_connection | |
| 17:55:13 | mriedem | *nova never calls | |
| 17:55:25 | mriedem | vivsoni__: so the explosion is likely on the cinder side | |
| 17:55:42 | mriedem | i.e. attachment_delete in cinder is assuming there will be a connector, and there isn't | |
| 17:57:56 | mriedem | vivsoni__: where is the error in the c-vol logs? http://54.201.44.218/38/527338/2/check/3par-fc-driver-master-client-pip-c8k01-dsvm/c156eb1/logs/screen-c-vol.txt.gz | |
| 17:58:27 | mriedem | ah here http://paste.openstack.org/show/628980/ | |
| 17:58:53 | mriedem | vivsoni__: open a cinder bug | |
| 17:59:32 | smcginnis | vivsoni__: That's a bug in the driver. It should handle that. | |
| 17:59:54 | mriedem | it's confusing how that ever worked | |
| 18:00:04 | melwitt | looks like nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSampleWithMultipleComputeOldAttachFlow is failing intermittently in the gate | |
| 18:00:25 | mriedem | smcginnis: vivsoni_: oh i see | |
| 18:00:27 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2004 | |
| 18:00:36 | mriedem | in the old flow, if we didn't have a connector, we just didn't call terminate_connection | |
| 18:00:56 | mriedem | melwitt: ok | |
| 18:02:10 | mriedem | smcginnis: which means other drivers could blow up | |
| 18:02:22 | mriedem | smcginnis: i'm thinking the volume manager's attachment_delete shouldn't call the driver if there is no connector | |
| 18:03:17 | vivsoni__ | mriedem, smcginnis: many 3dr paty CI has skipped those TCs | |
| 18:03:38 | mriedem | ok so i'll push a fix quick | |
| 18:03:55 | mriedem | vivsoni__: do you have a bug number? | |
| 18:04:28 | vivsoni__ | mriedem: No... not sure whether there is a bug raised | |
| 18:04:39 | mriedem | vivsoni__: ok, can you open a cinder bug then? | |
| 18:04:45 | vivsoni__ | sure | |
| 18:05:21 | vivsoni__ | please give a min.. will raised and let you know in a min or two | |
| 18:05:40 | smcginnis | We had discussed this in Cinder several months back. If there are any drivers that are not handling not having a connector, a bug should be filed against those drivers. | |
| 18:05:59 | smcginnis | But most (maybe some) should be handling it correctly. | |
| 18:07:19 | melwitt | I notice a lot of tracing of ComputeHostNotFound in gate n-cpu logs too | |
| 18:09:34 | mriedem | melwitt: that's been around for a long time, on startup | |
| 18:09:39 | mriedem | since ocata | |
| 18:09:51 | mriedem | happens when n-cpu starts before the host is discovered i think | |
| 18:10:37 | melwitt | oh. I thought it used to be a warning but not a backtrace | |
| 18:10:39 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Make request_spec.spec MediumText https://review.openstack.org/528012 | |
| 18:11:01 | mriedem | it's a warning + traceback | |
| 18:11:06 | mriedem | because of an oslo.log bug | |
| 18:11:19 | mriedem | see my failed attempts https://review.openstack.org/#/c/517102/ | |
| 18:14:12 | melwitt | looking at the oslo.log patch and trying to figure out, is the fix obvious or not -_- | |
| 18:15:22 | vivsoni__ | mriedem: https://bugs.launchpad.net/cinder/+bug/1738254 | |
| 18:15:24 | openstack | Launchpad bug 1738254 in Cinder "TEMPEST test case failed "tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON.test_detach_volume_shelved_or_offload_server"" [Undecided,New] | |