| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-09 | |||
| 18:58:25 | jmccarthy | lyarwood: \o lol | |
| 18:58:36 | jmccarthy | Zzz | |
| 19:53:11 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003 | |
| 19:55:11 | mriedem | dansmith: on this heal allocations CLI, i'm currently pulling instaces per cell and filtering by task_state=None since we don't want to mess with instances that are going through some task state transition, | |
| 19:55:33 | dansmith | yeah | |
| 19:55:37 | mriedem | however, for the purpose of reporting why we didn't create allocations for a given instance (like we do if it didn't have a host, e.g. shelved offloaded), | |
| 19:55:55 | mriedem | i'm thinking about removing that filter so we can just check instance.task_state in code and if not None, we log something and continue | |
| 19:56:30 | mriedem | because it would be weird to say we healed all instances when we don't know that we actually got all instances in a cell | |
| 19:56:50 | dansmith | okay | |
| 19:57:14 | dansmith | in many cases the instances with a task state outstanding will be a fraction of the total | |
| 19:57:22 | dansmith | not all cases, but probably reasonable as long as you're batching | |
| 19:58:18 | mriedem | by default we pull 50 instances per iteration | |
| 19:59:02 | dansmith | yeah, I mean I was stating that you were batching, not asking | |
| 20:01:13 | esberglu | efried: melwitt: mriedem: I've enabled run_validation and added scenario testing in PowerVM CI. This should unblock these three in the runway queue | |
| 20:01:16 | esberglu | Snapshot: https://review.openstack.org/#/c/543023/ | |
| 20:01:20 | esberglu | Localdisk: https://review.openstack.org/#/c/549053/ & https://review.openstack.org/#/c/549300/ | |
| 20:01:49 | esberglu | vSCSI is still blocked due to lack of CI, I have noted that on the etherpad | |
| 20:02:52 | dansmith | I thought it was going to make the runs take a long time? | |
| 20:02:58 | mriedem | ++lib/tempest:configure_tempest:429 [m iniset /opt/stack/tempest/etc/tempest.conf validation run_validation True | |
| 20:04:05 | openstackgerrit | Jay Pipes proposed openstack/nova master: add consumer.generation/create missing consumers https://review.openstack.org/557958 | |
| 20:04:06 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: add Project, User and Consumer objects https://review.openstack.org/565403 | |
| 20:04:07 | mriedem | looks like most of the scenario tests are skipped which is what we talked about http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html | |
| 20:04:07 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add create() methods to Project, User and Consumer https://review.openstack.org/565404 | |
| 20:04:08 | openstackgerrit | Jay Pipes proposed openstack/nova master: move consumer ensure to API layer https://review.openstack.org/565405 | |
| 20:04:09 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework allocation handler _allocations_dict() https://review.openstack.org/565407 | |
| 20:04:10 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add a microversion for consumer generation support https://review.openstack.org/565604 | |
| 20:05:04 | esberglu | mriedem: Most of the scenario tests require floating ips and/or volume support which we don't have. I've enabled test_server_basic_ops which is the test specifically called out as blocking | |
| 20:05:07 | mriedem | esberglu: dansmith: melwitt: yeah tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops is what we wanted | |
| 20:05:13 | mriedem | yup | |
| 20:05:14 | mriedem | lgtm | |
| 20:05:21 | mriedem | test_server_basic_ops does ssh | |
| 20:05:28 | esberglu | mriedem: Yep | |
| 20:05:41 | mriedem | https://github.com/openstack/tempest/blob/8d3f0bec6e5e7f396ca1ae9c89978e88ae4f6aeb/tempest/scenario/test_server_basic_ops.py#L30 | |
| 20:06:15 | esberglu | dansmith: test_server_basic_ops is takes a long time, but with tempest concurrency it doesn't actually do much to overall run time | |
| 20:06:53 | mriedem | 2018-05-09 12:32:43.192 23165 DEBUG tempest [-] compute-feature-enabled.config_drive = True log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2908 | |
| 20:06:57 | dansmith | well, right, it takes longer on other platforms too, but I thought that was the reason given for not enabling it | |
| 20:07:11 | mriedem | the volume stuff takes 10 minutes for them | |
| 20:07:31 | mriedem | and they don't have a good way to cleanup volume things in their backing storage | |
| 20:07:50 | dansmith | I don't understand what that means | |
| 20:08:00 | melwitt | which part of the log shows the pass/fail status of each test? I see in the console.txt it shows "will run" or "skip" | |
| 20:08:04 | dansmith | nor how it would be different for someone real trying to do this | |
| 20:08:18 | mriedem | melwitt: http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html | |
| 20:08:20 | esberglu | melwitt: powervm_os_ci.html in the top level dir | |
| 20:08:27 | melwitt | ah, thanks | |
| 20:09:18 | esberglu | dansmith: test_server_basic_ops doesn't do much to overall runtime. But running all of the volume scenario tests will | |
| 20:09:24 | mriedem | volume storage isn't on a local disk, i think is the main difference | |
| 20:09:35 | esberglu | Plus the cleanup issue mriedem mentioned | |
| 20:09:47 | esberglu | I'm working on a vSCSI CI job that can be run on demand | |
| 20:09:50 | dansmith | right, I don't understand the cleanup issue specifically | |
| 20:10:11 | mriedem | it can't handle the number of concurrent volume creates and deletes is my understanding | |
| 20:10:17 | mriedem | because it's enterprise storage :) | |
| 20:10:22 | dansmith | lol | |
| 20:11:11 | mriedem | i remember when i was running powerkvm ci and we had a similar setup, | |
| 20:11:26 | mriedem | it was bricked at least once per week, and that was serial runs twice per day | |
| 20:11:40 | dansmith | the original powervm ci was similar IIRC | |
| 20:11:46 | mriedem | *powerkvm ci for the ibm distro thing, not pkvm that ibm ltc runs | |
| 20:11:50 | esberglu | ouch | |
| 20:12:28 | mriedem | granted our cleanup bash scripts probably sucked | |
| 20:12:39 | mriedem | and i didn't know enough VIOS CLI to do much good, | |
| 20:12:58 | mriedem | but i remember having to sametime the lab admin once a week who eventually gave me creds to the IMM to just reboot the thing | |
| 20:13:39 | mriedem | i think he eventually pre-empted my sametimes because he knew we were breaking it | |
| 20:14:56 | mriedem | looking at these times, it's not much difference | |
| 20:14:57 | mriedem | IBM PowerVM CI (1 rechecks)1:33 PMnova-out-of-tree-pvmSUCCESS in 1h 27m 36snova-in-tree-pvmSUCCESS in 1h 31m 07s | |
| 20:15:17 | mriedem | efried: i think you can drop the -2 on this now https://review.openstack.org/#/c/543023/ | |
| 20:15:48 | efried | mriedem: Done. | |
| 20:36:19 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300 | |
| 20:40:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-manage placement heal_allocations CLI https://review.openstack.org/565886 | |
| 20:53:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deduplicate config/policy reference docs from main index https://review.openstack.org/564846 | |
| 20:53:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 20:53:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 20:53:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup placement policy generator docs https://review.openstack.org/565225 | |
| 21:01:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 21:01:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 21:27:32 | efried | mriedem: Is there any exception to the rule of proposing a thing in master and then backporting it? I.e. any exception process by which we can propose directly to stable without a corresponding master patch? | |
| 21:27:55 | edmondsw | if a bug only exists in a stable branch, then it'd be silly to merge something in master just to backport it to stable | |
| 21:28:12 | edmondsw | and then revert master | |
| 21:28:30 | edmondsw | so there must be cases where we make changes directly in a stable branch | |
| 21:39:00 | mriedem | efried: yes if the bug doesn't exist in master for the most part | |
| 21:39:22 | efried | mriedem: Okay, cool. edmondsw esberglu ^ | |
| 21:39:35 | mriedem | https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes | |
| 21:39:54 | efried | mriedem: We fixed it in pypowervm 1.1.15 which is now minimum in master nova, but we can't backport the req bump to stable, so we have to do a workaround in nova. | |
| 21:48:45 | mriedem | melwitt: you can post backports for this now https://review.openstack.org/#/c/555945/ | |
| 21:49:11 | mriedem | efried: i thought the breaking oslo lib change was reverted and blacklisted on stable? | |
| 21:50:18 | efried | mriedem: It wasn't reverted exactly; and the way in which it was "fixed" doesn't actually fix us. esberglu is posting the workaround; that commit message will have more details. | |
| 21:50:25 | efried | gory, gruesome details. | |
| 21:53:24 | mriedem | awesome | |
| 21:53:34 | mriedem | i'm going to delegate to tonyb and bauzas | |
| 21:54:18 | efried | mriedem: No worries, I was tapping you as the resident process nazi^Wguru to see if it was even legal to skip master. | |
| 21:55:32 | efried | drone cameras, baby. | |
| 21:59:24 | mriedem | zzzeek: ever seen "2018-05-09 16:29:33.278 11031 ERROR nova.api.openstack.extensions DBError: (pymysql.err.InternalError) (1, u"Can't create/write to file '/tmp/#sql_4038_1.MAI' (Errcode: 2)")" ? | |
| 21:59:28 | mriedem | in RHOSP 11 | |
| 21:59:53 | zzzeek | mriedem: there's a lot of files in /tmp/ and /tmp/ can run out of space or have fs problems | |
| 22:00:03 | zzzeek | mriedem: it also does this trick where it unlinks the file before using it | |
| 22:00:13 | zzzeek | so you can't see them with ls | |
| 22:01:07 | openstackgerrit | Merged openstack/nova stable/pike: Unmap compute nodes when deleting host mappings in delete cell operation https://review.openstack.org/553829 | |
| 22:01:13 | openstackgerrit | Merged openstack/nova master: Expose driver_block_device fields consistently https://review.openstack.org/528362 | |
| 22:01:20 | mriedem | ok; was just doing bug triage and someone reported that when listing instances | |
| 22:01:24 | openstackgerrit | Merged openstack/nova stable/queens: Fix detach_volume calls when rolling back a failed attach https://review.openstack.org/567232 | |
| 22:01:32 | openstackgerrit | Merged openstack/nova stable/queens: log stale allocations as WARNING instead of DEBUG https://review.openstack.org/567273 | |
| 22:01:55 | mriedem | i'll leave that info, thanks | |