| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-09 | |||
| 19:55:55 | mriedem | i'm thinking about removing that filter so we can just check instance.task_state in code and if not None, we log something and continue | |
| 19:56:30 | mriedem | because it would be weird to say we healed all instances when we don't know that we actually got all instances in a cell | |
| 19:56:50 | dansmith | okay | |
| 19:57:14 | dansmith | in many cases the instances with a task state outstanding will be a fraction of the total | |
| 19:57:22 | dansmith | not all cases, but probably reasonable as long as you're batching | |
| 19:58:18 | mriedem | by default we pull 50 instances per iteration | |
| 19:59:02 | dansmith | yeah, I mean I was stating that you were batching, not asking | |
| 20:01:13 | esberglu | efried: melwitt: mriedem: I've enabled run_validation and added scenario testing in PowerVM CI. This should unblock these three in the runway queue | |
| 20:01:16 | esberglu | Snapshot: https://review.openstack.org/#/c/543023/ | |
| 20:01:20 | esberglu | Localdisk: https://review.openstack.org/#/c/549053/ & https://review.openstack.org/#/c/549300/ | |
| 20:01:49 | esberglu | vSCSI is still blocked due to lack of CI, I have noted that on the etherpad | |
| 20:02:52 | dansmith | I thought it was going to make the runs take a long time? | |
| 20:02:58 | mriedem | ++lib/tempest:configure_tempest:429 [m iniset /opt/stack/tempest/etc/tempest.conf validation run_validation True | |
| 20:04:05 | openstackgerrit | Jay Pipes proposed openstack/nova master: add consumer.generation/create missing consumers https://review.openstack.org/557958 | |
| 20:04:06 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: add Project, User and Consumer objects https://review.openstack.org/565403 | |
| 20:04:07 | mriedem | looks like most of the scenario tests are skipped which is what we talked about http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html | |
| 20:04:07 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add create() methods to Project, User and Consumer https://review.openstack.org/565404 | |
| 20:04:08 | openstackgerrit | Jay Pipes proposed openstack/nova master: move consumer ensure to API layer https://review.openstack.org/565405 | |
| 20:04:09 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework allocation handler _allocations_dict() https://review.openstack.org/565407 | |
| 20:04:10 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add a microversion for consumer generation support https://review.openstack.org/565604 | |
| 20:05:04 | esberglu | mriedem: Most of the scenario tests require floating ips and/or volume support which we don't have. I've enabled test_server_basic_ops which is the test specifically called out as blocking | |
| 20:05:07 | mriedem | esberglu: dansmith: melwitt: yeah tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops is what we wanted | |
| 20:05:13 | mriedem | yup | |
| 20:05:14 | mriedem | lgtm | |
| 20:05:21 | mriedem | test_server_basic_ops does ssh | |
| 20:05:28 | esberglu | mriedem: Yep | |
| 20:05:41 | mriedem | https://github.com/openstack/tempest/blob/8d3f0bec6e5e7f396ca1ae9c89978e88ae4f6aeb/tempest/scenario/test_server_basic_ops.py#L30 | |
| 20:06:15 | esberglu | dansmith: test_server_basic_ops is takes a long time, but with tempest concurrency it doesn't actually do much to overall run time | |
| 20:06:53 | mriedem | 2018-05-09 12:32:43.192 23165 DEBUG tempest [-] compute-feature-enabled.config_drive = True log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2908 | |
| 20:06:57 | dansmith | well, right, it takes longer on other platforms too, but I thought that was the reason given for not enabling it | |
| 20:07:11 | mriedem | the volume stuff takes 10 minutes for them | |
| 20:07:31 | mriedem | and they don't have a good way to cleanup volume things in their backing storage | |
| 20:07:50 | dansmith | I don't understand what that means | |
| 20:08:00 | melwitt | which part of the log shows the pass/fail status of each test? I see in the console.txt it shows "will run" or "skip" | |
| 20:08:04 | dansmith | nor how it would be different for someone real trying to do this | |
| 20:08:18 | mriedem | melwitt: http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html | |
| 20:08:20 | esberglu | melwitt: powervm_os_ci.html in the top level dir | |
| 20:08:27 | melwitt | ah, thanks | |
| 20:09:18 | esberglu | dansmith: test_server_basic_ops doesn't do much to overall runtime. But running all of the volume scenario tests will | |
| 20:09:24 | mriedem | volume storage isn't on a local disk, i think is the main difference | |
| 20:09:35 | esberglu | Plus the cleanup issue mriedem mentioned | |
| 20:09:47 | esberglu | I'm working on a vSCSI CI job that can be run on demand | |
| 20:09:50 | dansmith | right, I don't understand the cleanup issue specifically | |
| 20:10:11 | mriedem | it can't handle the number of concurrent volume creates and deletes is my understanding | |
| 20:10:17 | mriedem | because it's enterprise storage :) | |
| 20:10:22 | dansmith | lol | |
| 20:11:11 | mriedem | i remember when i was running powerkvm ci and we had a similar setup, | |
| 20:11:26 | mriedem | it was bricked at least once per week, and that was serial runs twice per day | |
| 20:11:40 | dansmith | the original powervm ci was similar IIRC | |
| 20:11:46 | mriedem | *powerkvm ci for the ibm distro thing, not pkvm that ibm ltc runs | |
| 20:11:50 | esberglu | ouch | |
| 20:12:28 | mriedem | granted our cleanup bash scripts probably sucked | |
| 20:12:39 | mriedem | and i didn't know enough VIOS CLI to do much good, | |
| 20:12:58 | mriedem | but i remember having to sametime the lab admin once a week who eventually gave me creds to the IMM to just reboot the thing | |
| 20:13:39 | mriedem | i think he eventually pre-empted my sametimes because he knew we were breaking it | |
| 20:14:56 | mriedem | looking at these times, it's not much difference | |
| 20:14:57 | mriedem | IBM PowerVM CI (1 rechecks)1:33 PMnova-out-of-tree-pvmSUCCESS in 1h 27m 36snova-in-tree-pvmSUCCESS in 1h 31m 07s | |
| 20:15:17 | mriedem | efried: i think you can drop the -2 on this now https://review.openstack.org/#/c/543023/ | |
| 20:15:48 | efried | mriedem: Done. | |
| 20:36:19 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300 | |
| 20:40:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-manage placement heal_allocations CLI https://review.openstack.org/565886 | |
| 20:53:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deduplicate config/policy reference docs from main index https://review.openstack.org/564846 | |
| 20:53:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 20:53:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 20:53:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup placement policy generator docs https://review.openstack.org/565225 | |
| 21:01:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 21:01:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 21:27:32 | efried | mriedem: Is there any exception to the rule of proposing a thing in master and then backporting it? I.e. any exception process by which we can propose directly to stable without a corresponding master patch? | |
| 21:27:55 | edmondsw | if a bug only exists in a stable branch, then it'd be silly to merge something in master just to backport it to stable | |
| 21:28:12 | edmondsw | and then revert master | |
| 21:28:30 | edmondsw | so there must be cases where we make changes directly in a stable branch | |
| 21:39:00 | mriedem | efried: yes if the bug doesn't exist in master for the most part | |
| 21:39:22 | efried | mriedem: Okay, cool. edmondsw esberglu ^ | |
| 21:39:35 | mriedem | https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes | |
| 21:39:54 | efried | mriedem: We fixed it in pypowervm 1.1.15 which is now minimum in master nova, but we can't backport the req bump to stable, so we have to do a workaround in nova. | |
| 21:48:45 | mriedem | melwitt: you can post backports for this now https://review.openstack.org/#/c/555945/ | |
| 21:49:11 | mriedem | efried: i thought the breaking oslo lib change was reverted and blacklisted on stable? | |
| 21:50:18 | efried | mriedem: It wasn't reverted exactly; and the way in which it was "fixed" doesn't actually fix us. esberglu is posting the workaround; that commit message will have more details. | |
| 21:50:25 | efried | gory, gruesome details. | |
| 21:53:24 | mriedem | awesome | |
| 21:53:34 | mriedem | i'm going to delegate to tonyb and bauzas | |
| 21:54:18 | efried | mriedem: No worries, I was tapping you as the resident process nazi^Wguru to see if it was even legal to skip master. | |
| 21:55:32 | efried | drone cameras, baby. | |
| 21:59:24 | mriedem | zzzeek: ever seen "2018-05-09 16:29:33.278 11031 ERROR nova.api.openstack.extensions DBError: (pymysql.err.InternalError) (1, u"Can't create/write to file '/tmp/#sql_4038_1.MAI' (Errcode: 2)")" ? | |
| 21:59:28 | mriedem | in RHOSP 11 | |
| 21:59:53 | zzzeek | mriedem: there's a lot of files in /tmp/ and /tmp/ can run out of space or have fs problems | |
| 22:00:03 | zzzeek | mriedem: it also does this trick where it unlinks the file before using it | |
| 22:00:13 | zzzeek | so you can't see them with ls | |
| 22:01:07 | openstackgerrit | Merged openstack/nova stable/pike: Unmap compute nodes when deleting host mappings in delete cell operation https://review.openstack.org/553829 | |
| 22:01:13 | openstackgerrit | Merged openstack/nova master: Expose driver_block_device fields consistently https://review.openstack.org/528362 | |
| 22:01:20 | mriedem | ok; was just doing bug triage and someone reported that when listing instances | |
| 22:01:24 | openstackgerrit | Merged openstack/nova stable/queens: Fix detach_volume calls when rolling back a failed attach https://review.openstack.org/567232 | |
| 22:01:32 | openstackgerrit | Merged openstack/nova stable/queens: log stale allocations as WARNING instead of DEBUG https://review.openstack.org/567273 | |
| 22:01:55 | mriedem | i'll leave that info, thanks | |
| 22:04:25 | zzzeek | mriedem: i should find the command we used to see them, it was someting unpleasant | |
| 22:04:41 | zzzeek | oh it was lsof | |
| 22:04:52 | zzzeek | but , you see the same files over and over again | |
| 22:04:56 | openstackgerrit | Moshe Levi proposed openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640 | |
| 22:06:03 | zzzeek | mriedem: https://dev.mysql.com/doc/refman/8.0/en/temporary-files.html "MySQL arranges that temporary files are removed if mysqld is terminated. On platforms that support it (such as Unix), this is done by unlinking the file after opening it. The disadvantage of this is that the name does not appear in directory listings and you do not see a big temporary file that fills up the file system in which the temporary file directory is | |
| 22:06:03 | zzzeek | located. (In such cases, lsof +L1 may be helpful in identifying large files associated with mysqld.) " | |