Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-09
19:57:14 dansmith in many cases the instances with a task state outstanding will be a fraction of the total
19:57:22 dansmith not all cases, but probably reasonable as long as you're batching
19:58:18 mriedem by default we pull 50 instances per iteration
19:59:02 dansmith yeah, I mean I was stating that you were batching, not asking
20:01:13 esberglu efried: melwitt: mriedem: I've enabled run_validation and added scenario testing in PowerVM CI. This should unblock these three in the runway queue
20:01:16 esberglu Snapshot: https://review.openstack.org/#/c/543023/
20:01:20 esberglu Localdisk: https://review.openstack.org/#/c/549053/ & https://review.openstack.org/#/c/549300/
20:01:49 esberglu vSCSI is still blocked due to lack of CI, I have noted that on the etherpad
20:02:52 dansmith I thought it was going to make the runs take a long time?
20:02:58 mriedem ++lib/tempest:configure_tempest:429 [m iniset /opt/stack/tempest/etc/tempest.conf validation run_validation True
20:04:05 openstackgerrit Jay Pipes proposed openstack/nova master: add consumer.generation/create missing consumers https://review.openstack.org/557958
20:04:06 openstackgerrit Jay Pipes proposed openstack/nova master: placement: add Project, User and Consumer objects https://review.openstack.org/565403
20:04:07 openstackgerrit Jay Pipes proposed openstack/nova master: Add create() methods to Project, User and Consumer https://review.openstack.org/565404
20:04:07 mriedem looks like most of the scenario tests are skipped which is what we talked about http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html
20:04:08 openstackgerrit Jay Pipes proposed openstack/nova master: move consumer ensure to API layer https://review.openstack.org/565405
20:04:09 openstackgerrit Jay Pipes proposed openstack/nova master: rework allocation handler _allocations_dict() https://review.openstack.org/565407
20:04:10 openstackgerrit Jay Pipes proposed openstack/nova master: Add a microversion for consumer generation support https://review.openstack.org/565604
20:05:04 esberglu mriedem: Most of the scenario tests require floating ips and/or volume support which we don't have. I've enabled test_server_basic_ops which is the test specifically called out as blocking
20:05:07 mriedem esberglu: dansmith: melwitt: yeah tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops is what we wanted
20:05:13 mriedem yup
20:05:14 mriedem lgtm
20:05:21 mriedem test_server_basic_ops does ssh
20:05:28 esberglu mriedem: Yep
20:05:41 mriedem https://github.com/openstack/tempest/blob/8d3f0bec6e5e7f396ca1ae9c89978e88ae4f6aeb/tempest/scenario/test_server_basic_ops.py#L30
20:06:15 esberglu dansmith: test_server_basic_ops is takes a long time, but with tempest concurrency it doesn't actually do much to overall run time
20:06:53 mriedem 2018-05-09 12:32:43.192 23165 DEBUG tempest [-] compute-feature-enabled.config_drive = True log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2908
20:06:57 dansmith well, right, it takes longer on other platforms too, but I thought that was the reason given for not enabling it
20:07:11 mriedem the volume stuff takes 10 minutes for them
20:07:31 mriedem and they don't have a good way to cleanup volume things in their backing storage
20:07:50 dansmith I don't understand what that means
20:08:00 melwitt which part of the log shows the pass/fail status of each test? I see in the console.txt it shows "will run" or "skip"
20:08:04 dansmith nor how it would be different for someone real trying to do this
20:08:18 mriedem melwitt: http://184.172.12.213/23/543023/20/check/nova-in-tree-pvm/ab0b59a/powervm_os_ci.html
20:08:20 esberglu melwitt: powervm_os_ci.html in the top level dir
20:08:27 melwitt ah, thanks
20:09:18 esberglu dansmith: test_server_basic_ops doesn't do much to overall runtime. But running all of the volume scenario tests will
20:09:24 mriedem volume storage isn't on a local disk, i think is the main difference
20:09:35 esberglu Plus the cleanup issue mriedem mentioned
20:09:47 esberglu I'm working on a vSCSI CI job that can be run on demand
20:09:50 dansmith right, I don't understand the cleanup issue specifically
20:10:11 mriedem it can't handle the number of concurrent volume creates and deletes is my understanding
20:10:17 mriedem because it's enterprise storage :)
20:10:22 dansmith lol
20:11:11 mriedem i remember when i was running powerkvm ci and we had a similar setup,
20:11:26 mriedem it was bricked at least once per week, and that was serial runs twice per day
20:11:40 dansmith the original powervm ci was similar IIRC
20:11:46 mriedem *powerkvm ci for the ibm distro thing, not pkvm that ibm ltc runs
20:11:50 esberglu ouch
20:12:28 mriedem granted our cleanup bash scripts probably sucked
20:12:39 mriedem and i didn't know enough VIOS CLI to do much good,
20:12:58 mriedem but i remember having to sametime the lab admin once a week who eventually gave me creds to the IMM to just reboot the thing
20:13:39 mriedem i think he eventually pre-empted my sametimes because he knew we were breaking it
20:14:56 mriedem looking at these times, it's not much difference
20:14:57 mriedem IBM PowerVM CI (1 rechecks)1:33 PMnova-out-of-tree-pvmSUCCESS in 1h 27m 36snova-in-tree-pvmSUCCESS in 1h 31m 07s
20:15:17 mriedem efried: i think you can drop the -2 on this now https://review.openstack.org/#/c/543023/
20:15:48 efried mriedem: Done.
20:36:19 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300
20:40:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova-manage placement heal_allocations CLI https://review.openstack.org/565886
20:53:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425
20:53:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Deduplicate config/policy reference docs from main index https://review.openstack.org/564846
20:53:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578
20:53:38 openstackgerrit Matt Riedemann proposed openstack/nova master: Cleanup placement policy generator docs https://review.openstack.org/565225
21:01:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425
21:01:38 openstackgerrit Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578
21:27:32 efried mriedem: Is there any exception to the rule of proposing a thing in master and then backporting it? I.e. any exception process by which we can propose directly to stable without a corresponding master patch?
21:27:55 edmondsw if a bug only exists in a stable branch, then it'd be silly to merge something in master just to backport it to stable
21:28:12 edmondsw and then revert master
21:28:30 edmondsw so there must be cases where we make changes directly in a stable branch
21:39:00 mriedem efried: yes if the bug doesn't exist in master for the most part
21:39:22 efried mriedem: Okay, cool. edmondsw esberglu ^
21:39:35 mriedem https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes
21:39:54 efried mriedem: We fixed it in pypowervm 1.1.15 which is now minimum in master nova, but we can't backport the req bump to stable, so we have to do a workaround in nova.
21:48:45 mriedem melwitt: you can post backports for this now https://review.openstack.org/#/c/555945/
21:49:11 mriedem efried: i thought the breaking oslo lib change was reverted and blacklisted on stable?
21:50:18 efried mriedem: It wasn't reverted exactly; and the way in which it was "fixed" doesn't actually fix us. esberglu is posting the workaround; that commit message will have more details.
21:50:25 efried gory, gruesome details.
21:53:24 mriedem awesome
21:53:34 mriedem i'm going to delegate to tonyb and bauzas
21:54:18 efried mriedem: No worries, I was tapping you as the resident process nazi^Wguru to see if it was even legal to skip master.
21:55:32 efried drone cameras, baby.
21:59:24 mriedem zzzeek: ever seen "2018-05-09 16:29:33.278 11031 ERROR nova.api.openstack.extensions DBError: (pymysql.err.InternalError) (1, u"Can't create/write to file '/tmp/#sql_4038_1.MAI' (Errcode: 2)")" ?
21:59:28 mriedem in RHOSP 11
21:59:53 zzzeek mriedem: there's a lot of files in /tmp/ and /tmp/ can run out of space or have fs problems
22:00:03 zzzeek mriedem: it also does this trick where it unlinks the file before using it
22:00:13 zzzeek so you can't see them with ls
22:01:07 openstackgerrit Merged openstack/nova stable/pike: Unmap compute nodes when deleting host mappings in delete cell operation https://review.openstack.org/553829
22:01:13 openstackgerrit Merged openstack/nova master: Expose driver_block_device fields consistently https://review.openstack.org/528362
22:01:20 mriedem ok; was just doing bug triage and someone reported that when listing instances
22:01:24 openstackgerrit Merged openstack/nova stable/queens: Fix detach_volume calls when rolling back a failed attach https://review.openstack.org/567232
22:01:32 openstackgerrit Merged openstack/nova stable/queens: log stale allocations as WARNING instead of DEBUG https://review.openstack.org/567273
22:01:55 mriedem i'll leave that info, thanks
22:04:25 zzzeek mriedem: i should find the command we used to see them, it was someting unpleasant
22:04:41 zzzeek oh it was lsof
22:04:52 zzzeek but , you see the same files over and over again
22:04:56 openstackgerrit Moshe Levi proposed openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640
22:06:03 zzzeek located. (In such cases, lsof +L1 may be helpful in identifying large files associated with mysqld.) "
22:06:03 zzzeek mriedem: https://dev.mysql.com/doc/refman/8.0/en/temporary-files.html "MySQL arranges that temporary files are removed if mysqld is terminated. On platforms that support it (such as Unix), this is done by unlinking the file after opening it. The disadvantage of this is that the name does not appear in directory listings and you do not see a big temporary file that fills up the file system in which the temporary file directory is
22:07:47 openstack Launchpad bug 1769817 in OpenStack Compute (nova) "Ironic serial console doesn't get disabled when instance is deleted" [Undecided,Confirmed]
22:07:47 mriedem jroll: https://bugs.launchpad.net/nova/+bug/1769817 if you have any insight into that
22:20:03 mriedem jesus just triaged 2 separate duplicate bugs for this https://review.openstack.org/#/c/564336/ - people must really be getting screwed on that; dansmith - probably good to ^ get in

Earlier   Later