Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-16
22:42:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.opendev.org/541420
22:42:58 mriedem logan-: we're not sure, we need zzzeek's help for that
22:43:58 mriedem those are definitely related to the opportunistic db tests that fail,
22:44:07 mriedem the db is a random db name created by the test fixture,
22:44:12 mriedem openstack_citest is setup by a script on the node
22:44:32 mriedem https://github.com/openstack/nova/blob/master/tools/test-setup.sh does the mysql/postgresql setup
22:45:05 efried logan-: this might be what you were already looking at, but I did a compare of the mysql error log on a "good" run and a "bad" one, and didn't see significant differences.
22:45:21 efried logan-: see bottom-most comment on https://review.opendev.org/#/c/678051/
22:45:57 logan- yep, that's what i was wondering, thanks
22:47:05 efried although the bad one is taking way longer...
22:47:12 efried but dunno if that's cause or effect
22:47:39 logan- to answer the general question -- I don't know of anything significantly different on our nodes. they're pretty standard dual proc intel systems with ssds. I wonder if there's something in particular with this test that is wrecking the node somehow.
22:48:49 efried mhm, mriedem here's an interesting data point: in the "good" one, all those "abort" messages come through in under a minute. In the "bad" one, the first 16 or so are under a minute, then we start getting long delays.
22:49:32 efried now, IIUC, everything before those "abort" messages is just setup, and the "abort" things are happening during the actual meat of the test. So I guess that's not really surprising.
22:50:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.opendev.org/541420
22:53:37 efried It would be painful, but maybe if we knew what operations were being run, it would give us a clue as to what's happening when it starts to grind down.
22:54:08 efried anybody know how to turn on like trace logs to see the sql statements themselves as they're issued?
22:54:17 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Stop using volume_api.initialize_connection https://review.opendev.org/682508
22:55:20 mriedem efried: i want to say https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or
22:55:25 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace
22:57:54 alex_xu efried: https://review.opendev.org/682509
22:58:19 efried alex_xu: cool, thank you.
22:58:26 alex_xu efried: np!
22:58:36 efried alex_xu: since this is market-y, we probably don't want to mention the generic resource framework
22:58:49 alex_xu efried: ah, got it
22:58:59 efried that's a thing only devs would care about, at least until it reaps perf benefits via placement, which is a ways off.
22:59:14 efried alex_xu: perhaps kill that sentence and instead explain a bit more what vpmem is
22:59:15 alex_xu efried: sev mentioned that, that is why I follow
22:59:24 efried oh? looking...
23:00:05 efried their workloads when using the libvirt compute driver"
23:00:05 efried alex_xu: yeah, I'm looking for something corresponding to "to protect users against attackers or rogue administrators snooping on
23:00:16 alex_xu efried: got it
23:00:31 alex_xu mriedem: you are super fast
23:00:54 efried So like "Added $feature for $benefit"
23:01:56 mriedem "Added VPMEMs for Intel to sell hardware."
23:01:59 mriedem did it for you :)
23:02:29 sean-k-mooney add vpmem so the nsa can keep allyour data in "ram" to track you faster
23:02:41 mriedem "added vpmems so alex can continue working upstream"
23:03:07 mriedem i kid i kid
23:04:17 mriedem i think as long as you have HPC in there somewhere it will hit enough marketing bells to satisfy the entry
23:04:46 mriedem "Support virtual persistent memory devices for HPC workloads when using the libvirt driver.":
23:04:48 mriedem something like that
23:04:49 efried "edge"
23:04:51 sean-k-mooney the primary usecase is really something like "vPMEM support was intoduced to allow big data workloads to retain more data in persitent memroy reduceing the total cost of onership for big data clouds"
23:05:24 sean-k-mooney ya hpc is the other vertical that will likely use it most
23:05:30 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051
23:05:31 openstack bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251
23:05:37 efried mriedem, logan-: Howza ^
23:05:46 mriedem "other vertical"? someone get this guy a c-suite office
23:06:00 mriedem efried: it's dinner o'clock for me
23:06:03 efried me too
23:06:26 sean-k-mooney its tuesday for me so im going to sleep soon o/
23:07:35 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051
23:07:36 openstack bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251
23:11:19 alex_xu whatever that is worth to celebrate :)
#openstack-nova - 2019-09-17
00:16:05 alex_xu efried: updated https://review.opendev.org/682509
00:17:02 zzzeek logan-: not something ive seen before, https://dba.stackexchange.com/a/19139/81161 seems to have the best info
00:17:10 zzzeek max_packet_size
00:19:22 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511
00:20:04 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511
00:38:15 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511
00:45:31 openstackgerrit Brin Zhang proposed openstack/nova master: Add the maximum train flag in api version history https://review.opendev.org/682513
01:41:37 openstackgerrit Merged openstack/nova master: Follow up for the bandwidth series https://review.opendev.org/682389
01:41:44 openstackgerrit Merged openstack/nova master: Remove redundancies from AggregateRequestFiltersTest.setUp https://review.opendev.org/682475
01:42:11 openstackgerrit Merged openstack/nova master: doc: mark the max microversion for train https://review.opendev.org/681982
01:56:58 openstackgerrit hutianhao27 proposed openstack/nova master: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/682523
02:44:57 openstackgerrit ya.wang proposed openstack/nova master: Fix typo of cpu model when check CPU compatibility https://review.opendev.org/682267
05:16:56 openstackgerrit Merged openstack/nova master: Centralize volume create code during boot from volume https://review.opendev.org/682378
06:20:23 openstackgerrit Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969
07:14:00 bauzas good morning Nova
07:18:53 gibi good morning bauzas
07:43:22 bauzas gibi: any change you wanna stuff me in ?
07:59:37 gibi bauzas: nothing open at the moment I'm working on to fix mriedem's comments in the bug fix https://review.opendev.org/#/c/666857/
07:59:52 bauzas ack, saw iy
07:59:54 bauzas it*
08:00:01 gibi bauzas: I will ping you if that rework is ready
08:01:00 bauzas ++
08:01:56 gibi bauzas: do you have something I need to look at?
08:02:28 bauzas gibi: not atm, still fixing some tests for the placement audit
08:06:43 openstackgerrit Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765
08:06:43 openstackgerrit Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.opendev.org/648912
08:23:53 openstackgerrit Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857
09:41:12 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Remove stale BDMs on reserve_block_device_name failure https://review.opendev.org/682594
11:15:52 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM libvirt: extract _start_heterogeneous_computes https://review.opendev.org/682620
11:15:53 openstackgerrit Lee Yarwood proposed openstack/nova master: Add regression test for bug #1843639 https://review.opendev.org/682621
11:15:54 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/682622
11:15:54 openstack bug 1843639 in OpenStack Compute (nova) "libvirt: post_live_migration failures to disconnect volumes result in the rollback of live migrations" [Medium,Confirmed] https://launchpad.net/bugs/1843639
11:35:22 openstackgerrit Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857
11:57:49 openstackgerrit Ivaylo Mitev proposed openstack/nova master: VMware VMDK detach: get adapter type from instance VM https://review.opendev.org/653738
11:58:19 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/673790
11:58:20 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Methods to handle request for Secure Boot & non-Q35 machine types https://review.opendev.org/682628
11:58:20 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Add methods to detect firmware auto-selection and SB https://review.opendev.org/682627
12:01:38 gibi mriedem, bauzas: comments fixed i https://review.opendev.org/#/c/666857
12:05:41 stephenfin efried: Are we ever going to be able to merge a patch again? :D
12:10:46 kashyap (And remove needless compat code; we should have much less of it, though, as we've already cleaned up a good chunk of it a few months.)
12:17:25 efried stephenfin: appears this time it was mirror problems. But yeah, the gods really seem to hate that top pmem patch.
12:17:52 efried stephenfin: though mriedem and I fixed a regression from the forbidden aggs series which was causing a bunch of our failures.
12:18:11 stephenfin o rly? link?
12:18:12 efried now if we could just kill the innodb bug
12:18:28 efried stephenfin: https://review.opendev.org/#/c/682475/

Earlier   Later