| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-14 | |||
| 17:17:11 | bauzas | how could they ? | |
| 17:17:15 | openstackgerrit | Merged openstack/nova master: Fix disk size during live migration with disk over-commit https://review.openstack.org/524681 | |
| 17:17:21 | openstackgerrit | Merged openstack/nova stable/ocata: Only query BDMs once in API during rebuild https://review.openstack.org/525665 | |
| 17:17:34 | bauzas | do we expose the topology to the users ? hell no, of course | |
| 17:17:47 | bauzas | ah, that | |
| 17:18:04 | bauzas | well, two cases, and that requires looking at code | |
| 17:18:16 | bauzas | #1 either we fail and we reschedule to another host | |
| 17:18:21 | bauzas | which could eventually fail too | |
| 17:18:26 | bauzas | #2 or we make it full stop | |
| 17:18:33 | bauzas | and then we error out | |
| 17:18:45 | bauzas | either way, the user experience isn't terrible | |
| 17:18:51 | stephenfin | Oh, actually, #1 would actually happen | |
| 17:19:04 | stephenfin | because the NUMATopologyFilter (eventually) calls those functions | |
| 17:19:25 | bauzas | isn't that called by the compute service? | |
| 17:19:31 | stephenfin | That too, yes | |
| 17:19:36 | stephenfin | It's this call | |
| 17:20:34 | stephenfin | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L94-L98 | |
| 17:20:46 | bauzas | man, I got 3 days of legal and accouting training, but I don't know what hurts more : that or discussing PCI scheduling | |
| 17:20:57 | stephenfin | Hahaha | |
| 17:21:01 | mriedem | jaypipes: is an alter statement on a column type change idempotent? https://review.openstack.org/#/c/528012/1/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/051_request_spec_spec_mediumtext.py | |
| 17:21:04 | mriedem | if it's already that type? | |
| 17:21:13 | stephenfin | Don't worry - jaypipes and co are going to save us with nested-rps | |
| 17:21:26 | stephenfin | And if they don't - guaranteed job security :P | |
| 17:21:58 | bauzas | having a job security doesn't mean brain sanity | |
| 17:22:22 | mriedem | gd 11:30 already | |
| 17:22:44 | bauzas | 11:22 technically | |
| 17:22:53 | mriedem | not in minnesota | |
| 17:23:21 | bauzas | Minnesota is that better from other states that it saves 8 mins for every worker ? | |
| 17:23:29 | stephenfin | bauzas: But yeah, that call ends up here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1436 | |
| 17:23:36 | mriedem | bauzas: it is; that's why our taxes are high | |
| 17:24:02 | bauzas | mriedem: I'm sure we beat your state | |
| 17:24:04 | stephenfin | which then calls in here, where we _eventually_ raise the exception https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1513 | |
| 17:24:20 | mriedem | bauzas: sure, but, | |
| 17:24:31 | mriedem | we don't get 50% of the month off for PTO :P | |
| 17:25:04 | bauzas | jeez | |
| 17:25:07 | bauzas | touché | |
| 17:25:53 | bauzas | stephenfin: so, give me a sec to rephrase that | |
| 17:26:12 | bauzas | stephenfin: you mean that the policy check will be done primarly by the scheduler then ? | |
| 17:26:31 | bauzas | I need to rereview then the implications | |
| 17:26:35 | stephenfin | It'll be done twice - once by the scheduler and once when we're booting | |
| 17:26:45 | stephenfin | Because races are possible | |
| 17:26:49 | bauzas | yeah, but totally not at the same time :) | |
| 17:26:57 | stephenfin | Indeed :) | |
| 17:27:18 | stephenfin | So, per above, what a user will see is a failure to find suitable hosts | |
| 17:27:19 | bauzas | so, say we raise an exception, all of that stuff if called by the filter, right? | |
| 17:27:35 | stephenfin | All of what stuff? | |
| 17:27:35 | bauzas | s/if/is | |
| 17:28:37 | bauzas | stephenfin: https://review.openstack.org/#/c/527472/4/nova/virt/hardware.py@1197 is called eventually by the filter ? | |
| 17:28:56 | stephenfin | Yes | |
| 17:29:09 | stephenfin | Ditto for https://review.openstack.org/#/c/527472/4/nova/virt/hardware.py@1184 | |
| 17:29:44 | bauzas | stephenfin: so that worst than a reschedule | |
| 17:29:49 | bauzas | that's | |
| 17:30:05 | bauzas | stephenfin: if the filter raises an exception, the whole scheduling process fails | |
| 17:30:34 | bauzas | I need to test that, but I'm pretty it goes into ERROR/scheduling | |
| 17:30:46 | bauzas | oh man, fat fingers | |
| 17:31:43 | vivsoni__ | mriedem: Hi | |
| 17:32:37 | stephenfin | bauzas: Ack. I'm going to test it now myself with CPU policies | |
| 17:32:51 | bauzas | stephenfin: I need to bail out, family duty | |
| 17:32:56 | bauzas | ++ | |
| 17:33:10 | stephenfin | A-ok. I'm gone tomorrow but I'll be around Monday if you've questions | |
| 17:33:31 | mriedem | vivsoni__: hi | |
| 17:34:04 | bauzas | stephenfin: drop me a note about your findings and if that's not a full scheduling break, then +2 on the last PS | |
| 17:34:20 | stephenfin | bauzas: Spot on. Enjoy family time :) | |
| 17:34:28 | bauzas | if that's a full scheduling stop then I'm pretty -1 on the existing | |
| 17:34:47 | bauzas | or make the filter resilient to errors | |
| 17:35:04 | vivsoni__ | mriedem: Need your input in resolving tempest Test case "tempest.api.compute.volumes.test_attach_volume.AttachVolumeShelveTestJSON.test_detach_volume_shelved_or_offload_server" | |
| 17:35:25 | vivsoni__ | hpe ci is failing because of that | |
| 17:37:05 | openstackgerrit | Ade Lee proposed openstack/nova master: Add flag to make glance image verification optional https://review.openstack.org/528017 | |
| 17:39:32 | vivsoni__ | mriedem: In this TC 'connector', which is passed to cinder vendor driver's 'initialize_connection' & 'terminate_connection' is empty | |
| 17:43:21 | vivsoni__ | basically the driver's 'def initialize_connection' is not called, when 'attach_volume' is called https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L252 | |
| 17:43:59 | vivsoni__ | its failing here https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L253 | |
| 17:46:01 | mriedem | vivsoni__: that's because nova doesn't call initialize_connection anymore | |
| 17:46:04 | mriedem | it calls attachment_update | |
| 17:47:02 | vivsoni__ | this detach_volume(), internally calls to 'def terminate_connection()' of cinder driver's and failed when tries to get connector['host'] | |
| 17:48:25 | mriedem | vivsoni__: this would be easier if you could point me at CI job logs | |
| 17:48:27 | mriedem | with an actual failure | |
| 17:48:41 | mriedem | i am woefully behind on my todo list for today | |
| 17:49:07 | openstackgerrit | Merged openstack/nova stable/ocata: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/525666 | |
| 17:50:12 | vivsoni__ | mriedem: request you to please have a look at http://54.201.44.218/38/527338/2/check/3par-fc-driver-master-client-pip-c8k01-dsvm/c156eb1/ | |
| 17:53:18 | mriedem | vivsoni__: ok so it's attaching a volume to a shelved offloaded server, | |
| 17:53:24 | mriedem | so there is no host connector because there is no compute host for that server, | |
| 17:53:35 | mriedem | and then the test detaches the volume while the server is still shelved offloaded | |
| 17:53:44 | mriedem | and that's where it breaks right? | |
| 17:53:59 | vivsoni__ | mriedem: yes | |
| 17:54:09 | mriedem | ok that's right here v | |
| 17:54:10 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1994 | |
| 17:54:19 | mriedem | that's the new call to cinder in this case | |
| 17:54:31 | vivsoni__ | basically volume is not attach and tries to detaches and failed | |
| 17:54:56 | openstackgerrit | Merged openstack/nova stable/newton: Add regression test for rebuilding a volume-backed server https://review.openstack.org/526542 | |
| 17:55:09 | mriedem | vivsoni__: i'm not sure why this would be any different from a nova perspective - when attaching a volume to a shelved offloaded server, not never calls os-initialize_connection | |
| 17:55:13 | mriedem | *nova never calls | |
| 17:55:25 | mriedem | vivsoni__: so the explosion is likely on the cinder side | |
| 17:55:42 | mriedem | i.e. attachment_delete in cinder is assuming there will be a connector, and there isn't | |
| 17:57:56 | mriedem | vivsoni__: where is the error in the c-vol logs? http://54.201.44.218/38/527338/2/check/3par-fc-driver-master-client-pip-c8k01-dsvm/c156eb1/logs/screen-c-vol.txt.gz | |
| 17:58:27 | mriedem | ah here http://paste.openstack.org/show/628980/ | |
| 17:58:53 | mriedem | vivsoni__: open a cinder bug | |
| 17:59:32 | smcginnis | vivsoni__: That's a bug in the driver. It should handle that. | |
| 17:59:54 | mriedem | it's confusing how that ever worked | |
| 18:00:04 | melwitt | looks like nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSampleWithMultipleComputeOldAttachFlow is failing intermittently in the gate | |
| 18:00:25 | mriedem | smcginnis: vivsoni_: oh i see | |