| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-09 | |||
| 18:40:56 | jmccarthy | lyarwood: * s/aligh/align | |
| 18:42:54 | jmccarthy | lyarwood: maybe the log I sent earlier doesn't make senses .. it was grepped on req-138f35c8-ed21-4430-bf68-525574657926 | |
| 18:43:04 | lyarwood | jmccarthy: yeah there's an underlying libvirt issue here with the attach, I'm just trying to rule out some older known issues with passphrase length etc, for example https://bugzilla.redhat.com/show_bug.cgi?id=1447297 | |
| 18:43:04 | openstack | bugzilla.redhat.com bug 1447297 in libvirt "Unable to use LUKS passphrase that is exactly 16 bytes long" [Unspecified,Closed: errata] - Assigned to pkrempa | |
| 18:44:22 | lyarwood | jmccarthy: final ask, `openstack volume type show luks` just to confirm the key_size | |
| 18:44:33 | lyarwood | ^ I think that's the correct command at least | |
| 18:45:02 | lyarwood | https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/volume-type.html#volume-type-show yeah it is | |
| 18:45:32 | lyarwood | openstack volume type show --encryption-type luks | |
| 18:47:30 | jmccarthy | lyarwood: https://paste.fedoraproject.org/paste/34qbpIwDx867wt66JCnUlA | |
| 18:48:25 | lyarwood | jmccarthy: yeah 256 should be fine, odd. | |
| 18:48:36 | jmccarthy | lyarwood: I should be lioadm for the scsi_helper I think it is, right ? | |
| 18:48:37 | lyarwood | jmccarthy: is the same version of libvirt installed in both envs? | |
| 18:49:10 | lyarwood | jmccarthy: that really depends on the underlying env but that isn't the issue here | |
| 18:49:29 | lyarwood | jmccarthy: we can see the raw volume, crypsetup is able to format it etc | |
| 18:49:44 | lyarwood | jmccarthy: libvirt just borks when you try to attach it to the instance | |
| 18:50:00 | jmccarthy | lyarwood: I should have maybe mentioned earlier :/ one is tgdadm I think ? vs lioadm | |
| 18:50:47 | jmccarthy | lyarwood: Should be same version, have to confirm | |
| 18:51:02 | lyarwood | jmccarthy: I still can't see how that could cause this failure tbh | |
| 18:52:25 | jmccarthy | lyarwood: virsh --version says 3.9.0 on both | |
| 18:54:00 | jmccarthy | lyarwood: I know ? | |
| 18:54:24 | lyarwood | jmccarthy: kk, I'm out of ideas on the OpenStack / Nova side tbh, do you have libvirtd logs? | |
| 18:54:40 | lyarwood | jmccarthy: I need to drop in 5 btw before my wife kills me | |
| 18:55:10 | jmccarthy | lyarwood: Oh I did earlier lemme check again - they didn't say much - it's all good I'm fading also - I'll step through them tomorrow | |
| 18:56:09 | jmccarthy | lyarwood: Just more of this here `error : qemuMonitorJSONCheckError:389 : internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'` | |
| 18:56:31 | jmccarthy | lyarwood: Ok maybe I can get some more debug log out of that | |
| 18:56:55 | jmccarthy | lyarwood: Thanks for the help ! I thought maybe it was something obvious I was just missing, but I guess not | |
| 18:57:36 | lyarwood | jmccarthy: yeah increase the logging there and it might shed some light, iirc I've seen this before when the passphrase wouldn't unlock the volume, I just can't see how that's possible with the current flow tbh. | |
| 18:58:08 | jmccarthy | lyarwood: o/ | |
| 18:58:25 | jmccarthy | lyarwood: \o lol | |
| 18:58:36 | jmccarthy | Zzz | |
| 19:53:11 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003 | |
| 19:55:11 | mriedem | dansmith: on this heal allocations CLI, i'm currently pulling instaces per cell and filtering by task_state=None since we don't want to mess with instances that are going through some task state transition, | |
| 19:55:33 | dansmith | yeah | |
| 19:55:37 | mriedem | however, for the purpose of reporting why we didn't create allocations for a given instance (like we do if it didn't have a host, e.g. shelved offloaded), | |
| 19:55:55 | mriedem | i'm thinking about removing that filter so we can just check instance.task_state in code and if not None, we log something and continue | |
| 19:56:30 | mriedem | because it would be weird to say we healed all instances when we don't know that we actually got all instances in a cell | |
| 19:56:50 | dansmith | okay | |
| 19:57:14 | dansmith | in many cases the instances with a task state outstanding will be a fraction of the total | |
| 19:57:22 | dansmith | not all cases, but probably reasonable as long as you're batching | |
| 19:58:18 | mriedem | by default we pull 50 instances per iteration | |
| 19:59:02 | dansmith | yeah, I mean I was stating that you were batching, not asking | |
| 20:01:13 | esberglu | efried: melwitt: mriedem: I've enabled run_validation and added scenario testing in PowerVM CI. This should unblock these three in the runway queue | |
| 20:01:16 | esberglu | Snapshot: https://review.openstack.org/#/c/543023/ | |
| 20:01:20 | esberglu | Localdisk: https://review.openstack.org/#/c/549053/ & https://review.openstack.org/#/c/549300/ | |
| 20:01:49 | esberglu | vSCSI is still blocked due to lack of CI, I have noted that on the etherpad | |
| 20:02:52 | dansmith | I thought it was going to make the runs take a long time? | |
| 20:02:58 | mriedem | ++lib/tempest:configure_tempest:429 [m iniset /opt/stack/tempest/etc/tempest.conf validation run_validation True | |
| 20:04:05 | openstackgerrit | Jay Pipes proposed openstack/nova master: add consumer.generation/create missing consumers https://review.openstack.org/557958 | |
| 20:04:06 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: add Project, User and Consumer objects https://review.openstack.org/565403 | |
| 20:04:07 | mriedem | looks like most of the scenario tests are skipped which is what we talked about http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html | |
| 20:04:07 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add create() methods to Project, User and Consumer https://review.openstack.org/565404 | |
| 20:04:08 | openstackgerrit | Jay Pipes proposed openstack/nova master: move consumer ensure to API layer https://review.openstack.org/565405 | |
| 20:04:09 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework allocation handler _allocations_dict() https://review.openstack.org/565407 | |
| 20:04:10 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add a microversion for consumer generation support https://review.openstack.org/565604 | |
| 20:05:04 | esberglu | mriedem: Most of the scenario tests require floating ips and/or volume support which we don't have. I've enabled test_server_basic_ops which is the test specifically called out as blocking | |
| 20:05:07 | mriedem | esberglu: dansmith: melwitt: yeah tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops is what we wanted | |
| 20:05:13 | mriedem | yup | |
| 20:05:14 | mriedem | lgtm | |
| 20:05:21 | mriedem | test_server_basic_ops does ssh | |
| 20:05:28 | esberglu | mriedem: Yep | |
| 20:05:41 | mriedem | https://github.com/openstack/tempest/blob/8d3f0bec6e5e7f396ca1ae9c89978e88ae4f6aeb/tempest/scenario/test_server_basic_ops.py#L30 | |
| 20:06:15 | esberglu | dansmith: test_server_basic_ops is takes a long time, but with tempest concurrency it doesn't actually do much to overall run time | |
| 20:06:53 | mriedem | 2018-05-09 12:32:43.192 23165 DEBUG tempest [-] compute-feature-enabled.config_drive = True log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2908 | |
| 20:06:57 | dansmith | well, right, it takes longer on other platforms too, but I thought that was the reason given for not enabling it | |
| 20:07:11 | mriedem | the volume stuff takes 10 minutes for them | |
| 20:07:31 | mriedem | and they don't have a good way to cleanup volume things in their backing storage | |
| 20:07:50 | dansmith | I don't understand what that means | |
| 20:08:00 | melwitt | which part of the log shows the pass/fail status of each test? I see in the console.txt it shows "will run" or "skip" | |
| 20:08:04 | dansmith | nor how it would be different for someone real trying to do this | |
| 20:08:18 | mriedem | melwitt: http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html | |
| 20:08:20 | esberglu | melwitt: powervm_os_ci.html in the top level dir | |
| 20:08:27 | melwitt | ah, thanks | |
| 20:09:18 | esberglu | dansmith: test_server_basic_ops doesn't do much to overall runtime. But running all of the volume scenario tests will | |
| 20:09:24 | mriedem | volume storage isn't on a local disk, i think is the main difference | |
| 20:09:35 | esberglu | Plus the cleanup issue mriedem mentioned | |
| 20:09:47 | esberglu | I'm working on a vSCSI CI job that can be run on demand | |
| 20:09:50 | dansmith | right, I don't understand the cleanup issue specifically | |
| 20:10:11 | mriedem | it can't handle the number of concurrent volume creates and deletes is my understanding | |
| 20:10:17 | mriedem | because it's enterprise storage :) | |
| 20:10:22 | dansmith | lol | |
| 20:11:11 | mriedem | i remember when i was running powerkvm ci and we had a similar setup, | |
| 20:11:26 | mriedem | it was bricked at least once per week, and that was serial runs twice per day | |
| 20:11:40 | dansmith | the original powervm ci was similar IIRC | |
| 20:11:46 | mriedem | *powerkvm ci for the ibm distro thing, not pkvm that ibm ltc runs | |
| 20:11:50 | esberglu | ouch | |
| 20:12:28 | mriedem | granted our cleanup bash scripts probably sucked | |
| 20:12:39 | mriedem | and i didn't know enough VIOS CLI to do much good, | |
| 20:12:58 | mriedem | but i remember having to sametime the lab admin once a week who eventually gave me creds to the IMM to just reboot the thing | |
| 20:13:39 | mriedem | i think he eventually pre-empted my sametimes because he knew we were breaking it | |
| 20:14:56 | mriedem | looking at these times, it's not much difference | |
| 20:14:57 | mriedem | IBM PowerVM CI (1 rechecks)1:33 PMnova-out-of-tree-pvmSUCCESS in 1h 27m 36snova-in-tree-pvmSUCCESS in 1h 31m 07s | |
| 20:15:17 | mriedem | efried: i think you can drop the -2 on this now https://review.openstack.org/#/c/543023/ | |
| 20:15:48 | efried | mriedem: Done. | |
| 20:36:19 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300 | |
| 20:40:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-manage placement heal_allocations CLI https://review.openstack.org/565886 | |
| 20:53:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deduplicate config/policy reference docs from main index https://review.openstack.org/564846 | |
| 20:53:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 20:53:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 20:53:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup placement policy generator docs https://review.openstack.org/565225 | |
| 21:01:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |