| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-14 | |||
| 17:33:10 | stephenfin | A-ok. I'm gone tomorrow but I'll be around Monday if you've questions | |
| 17:33:31 | mriedem | vivsoni__: hi | |
| 17:34:04 | bauzas | stephenfin: drop me a note about your findings and if that's not a full scheduling break, then +2 on the last PS | |
| 17:34:20 | stephenfin | bauzas: Spot on. Enjoy family time :) | |
| 17:34:28 | bauzas | if that's a full scheduling stop then I'm pretty -1 on the existing | |
| 17:34:47 | bauzas | or make the filter resilient to errors | |
| 17:35:04 | vivsoni__ | mriedem: Need your input in resolving tempest Test case "tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON.test_detach_volume_shelved_or_offload_server" | |
| 17:35:25 | vivsoni__ | hpe ci is failing because of that | |
| 17:37:05 | openstackgerrit | Ade Lee proposed openstack/nova master: Add flag to make glance image verification optional https://review.openstack.org/528017 | |
| 17:39:32 | vivsoni__ | mriedem: In this TC 'connector', which is passed to cinder vendor driver's 'initialize_connection' & 'terminate_connection' is empty | |
| 17:43:21 | vivsoni__ | basically the driver's 'def initialize_connection' is not called, when 'attach_volume' is called https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L252 | |
| 17:43:59 | vivsoni__ | its failing here https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L253 | |
| 17:46:01 | mriedem | vivsoni__: that's because nova doesn't call initialize_connection anymore | |
| 17:46:04 | mriedem | it calls attachment_update | |
| 17:47:02 | vivsoni__ | this detach_volume(), internally calls to 'def terminate_connection()' of cinder driver's and failed when tries to get connector['host'] | |
| 17:48:25 | mriedem | vivsoni__: this would be easier if you could point me at CI job logs | |
| 17:48:27 | mriedem | with an actual failure | |
| 17:48:41 | mriedem | i am woefully behind on my todo list for today | |
| 17:49:07 | openstackgerrit | Merged openstack/nova stable/ocata: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/525666 | |
| 17:50:12 | vivsoni__ | mriedem: request you to please have a look at http://54.201.44.218/38/527338/2/check/3par-fc-driver-master-client-pip-c8k01-dsvm/c156eb1/ | |
| 17:53:18 | mriedem | vivsoni__: ok so it's attaching a volume to a shelved offloaded server, | |
| 17:53:24 | mriedem | so there is no host connector because there is no compute host for that server, | |
| 17:53:35 | mriedem | and then the test detaches the volume while the server is still shelved offloaded | |
| 17:53:44 | mriedem | and that's where it breaks right? | |
| 17:53:59 | vivsoni__ | mriedem: yes | |
| 17:54:09 | mriedem | ok that's right here v | |
| 17:54:10 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1994 | |
| 17:54:19 | mriedem | that's the new call to cinder in this case | |
| 17:54:31 | vivsoni__ | basically volume is not attach and tries to detaches and failed | |
| 17:54:56 | openstackgerrit | Merged openstack/nova stable/newton: Add regression test for rebuilding a volume-backed server https://review.openstack.org/526542 | |
| 17:55:09 | mriedem | vivsoni__: i'm not sure why this would be any different from a nova perspective - when attaching a volume to a shelved offloaded server, not never calls os-initialize_connection | |
| 17:55:13 | mriedem | *nova never calls | |
| 17:55:25 | mriedem | vivsoni__: so the explosion is likely on the cinder side | |
| 17:55:42 | mriedem | i.e. attachment_delete in cinder is assuming there will be a connector, and there isn't | |
| 17:57:56 | mriedem | vivsoni__: where is the error in the c-vol logs? http://54.201.44.218/38/527338/2/check/3par-fc-driver-master-client-pip-c8k01-dsvm/c156eb1/logs/screen-c-vol.txt.gz | |
| 17:58:27 | mriedem | ah here http://paste.openstack.org/show/628980/ | |
| 17:58:53 | mriedem | vivsoni__: open a cinder bug | |
| 17:59:32 | smcginnis | vivsoni__: That's a bug in the driver. It should handle that. | |
| 17:59:54 | mriedem | it's confusing how that ever worked | |
| 18:00:04 | melwitt | looks like nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSampleWithMultipleComputeOldAttachFlow is failing intermittently in the gate | |
| 18:00:25 | mriedem | smcginnis: vivsoni_: oh i see | |
| 18:00:27 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2004 | |
| 18:00:36 | mriedem | in the old flow, if we didn't have a connector, we just didn't call terminate_connection | |
| 18:00:56 | mriedem | melwitt: ok | |
| 18:02:10 | mriedem | smcginnis: which means other drivers could blow up | |
| 18:02:22 | mriedem | smcginnis: i'm thinking the volume manager's attachment_delete shouldn't call the driver if there is no connector | |
| 18:03:17 | vivsoni__ | mriedem, smcginnis: many 3dr paty CI has skipped those TCs | |
| 18:03:38 | mriedem | ok so i'll push a fix quick | |
| 18:03:55 | mriedem | vivsoni__: do you have a bug number? | |
| 18:04:28 | vivsoni__ | mriedem: No... not sure whether there is a bug raised | |
| 18:04:39 | mriedem | vivsoni__: ok, can you open a cinder bug then? | |
| 18:04:45 | vivsoni__ | sure | |
| 18:05:21 | vivsoni__ | please give a min.. will raised and let you know in a min or two | |
| 18:05:40 | smcginnis | We had discussed this in Cinder several months back. If there are any drivers that are not handling not having a connector, a bug should be filed against those drivers. | |
| 18:05:59 | smcginnis | But most (maybe some) should be handling it correctly. | |
| 18:07:19 | melwitt | I notice a lot of tracing of ComputeHostNotFound in gate n-cpu logs too | |
| 18:09:34 | mriedem | melwitt: that's been around for a long time, on startup | |
| 18:09:39 | mriedem | since ocata | |
| 18:09:51 | mriedem | happens when n-cpu starts before the host is discovered i think | |
| 18:10:37 | melwitt | oh. I thought it used to be a warning but not a backtrace | |
| 18:10:39 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Make request_spec.spec MediumText https://review.openstack.org/528012 | |
| 18:11:01 | mriedem | it's a warning + traceback | |
| 18:11:06 | mriedem | because of an oslo.log bug | |
| 18:11:19 | mriedem | see my failed attempts https://review.openstack.org/#/c/517102/ | |
| 18:14:12 | melwitt | looking at the oslo.log patch and trying to figure out, is the fix obvious or not -_- | |
| 18:15:22 | vivsoni__ | mriedem: https://bugs.launchpad.net/cinder/+bug/1738254 | |
| 18:15:24 | openstack | Launchpad bug 1738254 in Cinder "TEMPEST test case failed "tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON.test_detach_volume_shelved_or_offload_server"" [Undecided,New] | |
| 18:15:47 | vivsoni__ | mriedem: i have raised a cinder bug against that tempest failure | |
| 18:17:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCI NUMA policies https://review.openstack.org/527472 | |
| 18:17:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Document PCI NUMA affinity policy https://review.openstack.org/528011 | |
| 18:17:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Modify signature of _filter_non_requested_pfs https://review.openstack.org/527473 | |
| 18:17:58 | melwitt | I think I can propose something | |
| 18:18:30 | stephenfin | bauzas: For when you're back online, that image-flavor conflict exception gets bubbled all the way back to the user's terminal via a HTTP 400. I think we're good | |
| 18:29:35 | stephenfin | melwitt: I'm not about tomorrow but if you have a chance to review [1] before Monday, I'd be eternally grateful [1] https://review.openstack.org/#/q/topic:bp/websocket-proxy-to-host-security+status:open | |
| 18:30:21 | stephenfin | melwitt: I think jianghuaw_'s comments are valid but something that can be addressed in a follow-up (I don't know how much work 3.8 -> 3.3 conversion would be) | |
| 18:30:45 | melwitt | stephenfin: I've been looking at it and it looks overall good to me, but I saw recently someone -1ed it saying it doesn't work with xen? okay | |
| 18:47:32 | pprokop | 1 | |
| 18:55:44 | melwitt | mriedem: let's see how this goes https://review.openstack.org/#/c/528036 | |
| 18:56:24 | mriedem | cool | |
| 19:05:15 | cfriesen | when dealing with quotas, why do we distinguish between "in use" and "reserved"? | |
| 19:06:33 | melwitt | because the old system had an initial "reserve" step that happened before switching it from "reserved" -> "in use" | |
| 19:07:14 | melwitt | reserve() reserved it and commit() changed it from reserved to in use | |
| 19:08:53 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 19:08:53 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 19:09:15 | cfriesen | melwitt: thanks...so in pike would we ever see a nonzero reserved value? | |
| 19:11:25 | melwitt | cfriesen: right, no more reserve step so reserved always zero | |
| 19:12:46 | cfriesen | melwitt: great, thanks | |
| 20:45:05 | openstackgerrit | melanie witt proposed openstack/nova master: live-mig: keep disk device address same https://review.openstack.org/518022 | |
| 21:27:15 | mriedem | can someone hit this? https://review.openstack.org/#/c/527878/ - unblocks the 2.58 api change and is needed for the common server create api samples after 2.57 where 'personality' is not an allowed request parameter | |
| 21:27:34 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Implement set_and_clear_allocations in report client https://review.openstack.org/528089 | |
| 21:28:17 | cdent | dansmith. mriedem that ^ is a start on POST to /allocations for migration stuff. It's just a spike at this point but I figured I'd push it up for early "is this what we were thinking style review". Also I needed to do something besides talk. | |
| 21:28:39 | mriedem | heh, cool. thanks. | |
| 21:28:45 | mriedem | that was on my list as well | |
| 21:30:01 | dansmith | sweet | |
| 21:30:20 | dansmith | I was literally lying in bed last night thinking I needed to ask if you were interested in doing that | |
| 21:30:21 | dansmith | and then forgot | |
| 21:30:26 | cdent | it appears to "work". the test_servers functional tests pass, and I assume it exercised by that | |
| 21:31:12 | cdent | actually, dansmith, I was standing in your room last night, watching you sleep | |
| 21:31:12 | tonyb | mriedem: Thanks for the update. I'll try again to look at thos backports. | |
| 21:31:41 | cdent | heh | |