Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-09
20:09:50 dansmith right, I don't understand the cleanup issue specifically
20:10:11 mriedem it can't handle the number of concurrent volume creates and deletes is my understanding
20:10:17 mriedem because it's enterprise storage :)
20:10:22 dansmith lol
20:11:11 mriedem i remember when i was running powerkvm ci and we had a similar setup,
20:11:26 mriedem it was bricked at least once per week, and that was serial runs twice per day
20:11:40 dansmith the original powervm ci was similar IIRC
20:11:46 mriedem *powerkvm ci for the ibm distro thing, not pkvm that ibm ltc runs
20:11:50 esberglu ouch
20:12:28 mriedem granted our cleanup bash scripts probably sucked
20:12:39 mriedem and i didn't know enough VIOS CLI to do much good,
20:12:58 mriedem but i remember having to sametime the lab admin once a week who eventually gave me creds to the IMM to just reboot the thing
20:13:39 mriedem i think he eventually pre-empted my sametimes because he knew we were breaking it
20:14:56 mriedem looking at these times, it's not much difference
20:14:57 mriedem IBM PowerVM CI (1 rechecks)1:33 PMnova-out-of-tree-pvmSUCCESS in 1h 27m 36snova-in-tree-pvmSUCCESS in 1h 31m 07s
20:15:17 mriedem efried: i think you can drop the -2 on this now https://review.openstack.org/#/c/543023/
20:15:48 efried mriedem: Done.
20:36:19 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300
20:40:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Add nova-manage placement heal_allocations CLI https://review.openstack.org/565886
20:53:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425
20:53:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Deduplicate config/policy reference docs from main index https://review.openstack.org/564846
20:53:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578
20:53:38 openstackgerrit Matt Riedemann proposed openstack/nova master: Cleanup placement policy generator docs https://review.openstack.org/565225
21:01:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425
21:01:38 openstackgerrit Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578
21:27:32 efried mriedem: Is there any exception to the rule of proposing a thing in master and then backporting it? I.e. any exception process by which we can propose directly to stable without a corresponding master patch?
21:27:55 edmondsw if a bug only exists in a stable branch, then it'd be silly to merge something in master just to backport it to stable
21:28:12 edmondsw and then revert master
21:28:30 edmondsw so there must be cases where we make changes directly in a stable branch
21:39:00 mriedem efried: yes if the bug doesn't exist in master for the most part
21:39:22 efried mriedem: Okay, cool. edmondsw esberglu ^
21:39:35 mriedem https://docs.openstack.org/project-team-guide/stable-branches.html#appropriate-fixes
21:39:54 efried mriedem: We fixed it in pypowervm 1.1.15 which is now minimum in master nova, but we can't backport the req bump to stable, so we have to do a workaround in nova.
21:48:45 mriedem melwitt: you can post backports for this now https://review.openstack.org/#/c/555945/
21:49:11 mriedem efried: i thought the breaking oslo lib change was reverted and blacklisted on stable?
21:50:18 efried mriedem: It wasn't reverted exactly; and the way in which it was "fixed" doesn't actually fix us. esberglu is posting the workaround; that commit message will have more details.
21:50:25 efried gory, gruesome details.
21:53:24 mriedem awesome
21:53:34 mriedem i'm going to delegate to tonyb and bauzas
21:54:18 efried mriedem: No worries, I was tapping you as the resident process nazi^Wguru to see if it was even legal to skip master.
21:55:32 efried drone cameras, baby.
21:59:24 mriedem zzzeek: ever seen "2018-05-09 16:29:33.278 11031 ERROR nova.api.openstack.extensions DBError: (pymysql.err.InternalError) (1, u"Can't create/write to file '/tmp/#sql_4038_1.MAI' (Errcode: 2)")" ?
21:59:28 mriedem in RHOSP 11
21:59:53 zzzeek mriedem: there's a lot of files in /tmp/ and /tmp/ can run out of space or have fs problems
22:00:03 zzzeek mriedem: it also does this trick where it unlinks the file before using it
22:00:13 zzzeek so you can't see them with ls
22:01:07 openstackgerrit Merged openstack/nova stable/pike: Unmap compute nodes when deleting host mappings in delete cell operation https://review.openstack.org/553829
22:01:13 openstackgerrit Merged openstack/nova master: Expose driver_block_device fields consistently https://review.openstack.org/528362
22:01:20 mriedem ok; was just doing bug triage and someone reported that when listing instances
22:01:24 openstackgerrit Merged openstack/nova stable/queens: Fix detach_volume calls when rolling back a failed attach https://review.openstack.org/567232
22:01:32 openstackgerrit Merged openstack/nova stable/queens: log stale allocations as WARNING instead of DEBUG https://review.openstack.org/567273
22:01:55 mriedem i'll leave that info, thanks
22:04:25 zzzeek mriedem: i should find the command we used to see them, it was someting unpleasant
22:04:41 zzzeek oh it was lsof
22:04:52 zzzeek but , you see the same files over and over again
22:04:56 openstackgerrit Moshe Levi proposed openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640
22:06:03 zzzeek located. (In such cases, lsof +L1 may be helpful in identifying large files associated with mysqld.) "
22:06:03 zzzeek mriedem: https://dev.mysql.com/doc/refman/8.0/en/temporary-files.html "MySQL arranges that temporary files are removed if mysqld is terminated. On platforms that support it (such as Unix), this is done by unlinking the file after opening it. The disadvantage of this is that the name does not appear in directory listings and you do not see a big temporary file that fills up the file system in which the temporary file directory is
22:07:47 openstack Launchpad bug 1769817 in OpenStack Compute (nova) "Ironic serial console doesn't get disabled when instance is deleted" [Undecided,Confirmed]
22:07:47 mriedem jroll: https://bugs.launchpad.net/nova/+bug/1769817 if you have any insight into that
22:20:03 mriedem jesus just triaged 2 separate duplicate bugs for this https://review.openstack.org/#/c/564336/ - people must really be getting screwed on that; dansmith - probably good to ^ get in
22:20:10 mriedem err https://review.openstack.org/#/c/564336/
22:29:09 openstack Launchpad bug 1769025 in OpenStack Compute (nova) "OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object error is thrown for ironic deploy in Pike" [Undecided,New]
22:29:09 mriedem does this ring a bell? https://bugs.launchpad.net/nova/+bug/1769025
22:29:15 mriedem that's on pike
22:35:33 openstackgerrit Jay Pipes proposed openstack/nova master: Add a microversion for consumer generation support https://review.openstack.org/565604
22:39:24 openstack Launchpad bug 1768876 in OpenStack Compute (nova) "Old instances can get AVZ from metadata" [Undecided,New] - Assigned to Surya Seetharaman (tssurya)
22:39:24 mriedem ooo upcalls https://bugs.launchpad.net/nova/+bug/1768876
22:41:01 mriedem so apparently the metadata-api used to do an up-call to the API DB to get the AZ for an instance, but in pike we started storing the AZ directly on the instance rather than the dumb default 'nova' AZ,
22:41:15 mriedem but for any instances created before pike, they aren't going to have that, so they needed the up-call
22:43:52 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Granular requests to get_allocation_candidates https://review.openstack.org/515811
22:48:18 openstackgerrit Merged openstack/nova stable/queens: Make association_refresh configurable https://review.openstack.org/566288
23:00:21 mriedem melwitt: everything is ready for queens except https://review.openstack.org/#/c/566367/ which is still on master
23:00:30 mriedem will redouble my bugging efforts in the wee morn
23:00:45 mriedem same for pike and ocata actually
23:02:48 tonyb efried, esberglu: Add me as a reviewer on the patch and I'll review it with my stable hat
23:25:05 openstackgerrit Merged openstack/nova master: Remove mox in test_xenapi.py (1) https://review.openstack.org/564639
23:51:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_xenapi.py (2) https://review.openstack.org/564643
23:51:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in test_xenapi.py (3) https://review.openstack.org/564645
#openstack-nova - 2018-05-10
00:29:19 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Return nested providers in get_by_request https://review.openstack.org/567113
00:29:20 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Add traits check in nested provider cadidates https://review.openstack.org/567150
00:29:21 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix broken allocation candidates with traits https://review.openstack.org/567151
00:56:37 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Add traits check in nested provider cadidates https://review.openstack.org/567150
00:56:38 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix broken allocation candidates with traits https://review.openstack.org/567151
01:30:33 openstackgerrit Merged openstack/nova master: Convert configdrive to use processutils. https://review.openstack.org/565723
01:43:47 openstackgerrit Erik Olof Gunnar Andersson proposed openstack/nova master: Remove remaning log translation in scheduler https://review.openstack.org/567455
02:56:08 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: Initial change set of z/VM driver https://review.openstack.org/523387
02:56:09 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: Spawn and destroy function of z/VM driver https://review.openstack.org/527658
02:56:10 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: add snapshot function https://review.openstack.org/534240
02:56:11 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: add power actions https://review.openstack.org/543340
02:56:12 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: add get console output https://review.openstack.org/543344
02:56:13 openstackgerrit Merged openstack/nova master: Add tests for alloc cands with poor local disk https://review.openstack.org/566842
04:39:53 openstackgerrit Merged openstack/nova master: Migrate tempest-dsvm-multinode-live-migration job in-tree https://review.openstack.org/555945
05:45:35 openstackgerrit Tsuyoshi Nagata proposed openstack/nova master: nova improvement of maximum attach volumes more than 26 vols https://review.openstack.org/567472
05:57:08 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: Initial change set of z/VM driver https://review.openstack.org/523387
05:57:09 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: Spawn and destroy function of z/VM driver https://review.openstack.org/527658
05:57:10 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: add snapshot function https://review.openstack.org/534240
05:57:11 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: add power actions https://review.openstack.org/543340
05:57:12 openstackgerrit jichenjc proposed openstack/nova master: z/VM Driver: add get console output https://review.openstack.org/543344
06:19:32 openstackgerrit Hamdy Khader proposed openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640

Earlier   Later