| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-16 | |||
| 22:44:07 | mriedem | the db is a random db name created by the test fixture, | |
| 22:44:12 | mriedem | openstack_citest is setup by a script on the node | |
| 22:44:32 | mriedem | https://github.com/openstack/nova/blob/master/tools/test-setup.sh does the mysql/postgresql setup | |
| 22:45:05 | efried | logan-: this might be what you were already looking at, but I did a compare of the mysql error log on a "good" run and a "bad" one, and didn't see significant differences. | |
| 22:45:21 | efried | logan-: see bottom-most comment on https://review.opendev.org/#/c/678051/ | |
| 22:45:57 | logan- | yep, that's what i was wondering, thanks | |
| 22:47:05 | efried | although the bad one is taking way longer... | |
| 22:47:12 | efried | but dunno if that's cause or effect | |
| 22:47:39 | logan- | to answer the general question -- I don't know of anything significantly different on our nodes. they're pretty standard dual proc intel systems with ssds. I wonder if there's something in particular with this test that is wrecking the node somehow. | |
| 22:48:49 | efried | mhm, mriedem here's an interesting data point: in the "good" one, all those "abort" messages come through in under a minute. In the "bad" one, the first 16 or so are under a minute, then we start getting long delays. | |
| 22:49:32 | efried | now, IIUC, everything before those "abort" messages is just setup, and the "abort" things are happening during the actual meat of the test. So I guess that's not really surprising. | |
| 22:50:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.opendev.org/541420 | |
| 22:53:37 | efried | It would be painful, but maybe if we knew what operations were being run, it would give us a clue as to what's happening when it starts to grind down. | |
| 22:54:08 | efried | anybody know how to turn on like trace logs to see the sql statements themselves as they're issued? | |
| 22:54:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Stop using volume_api.initialize_connection https://review.opendev.org/682508 | |
| 22:55:20 | mriedem | efried: i want to say https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or | |
| 22:55:25 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace | |
| 22:57:54 | alex_xu | efried: https://review.opendev.org/682509 | |
| 22:58:19 | efried | alex_xu: cool, thank you. | |
| 22:58:26 | alex_xu | efried: np! | |
| 22:58:36 | efried | alex_xu: since this is market-y, we probably don't want to mention the generic resource framework | |
| 22:58:49 | alex_xu | efried: ah, got it | |
| 22:58:59 | efried | that's a thing only devs would care about, at least until it reaps perf benefits via placement, which is a ways off. | |
| 22:59:14 | efried | alex_xu: perhaps kill that sentence and instead explain a bit more what vpmem is | |
| 22:59:15 | alex_xu | efried: sev mentioned that, that is why I follow | |
| 22:59:24 | efried | oh? looking... | |
| 23:00:05 | efried | their workloads when using the libvirt compute driver" | |
| 23:00:05 | efried | alex_xu: yeah, I'm looking for something corresponding to "to protect users against attackers or rogue administrators snooping on | |
| 23:00:16 | alex_xu | efried: got it | |
| 23:00:31 | alex_xu | mriedem: you are super fast | |
| 23:00:54 | efried | So like "Added $feature for $benefit" | |
| 23:01:56 | mriedem | "Added VPMEMs for Intel to sell hardware." | |
| 23:01:59 | mriedem | did it for you :) | |
| 23:02:29 | sean-k-mooney | add vpmem so the nsa can keep allyour data in "ram" to track you faster | |
| 23:02:41 | mriedem | "added vpmems so alex can continue working upstream" | |
| 23:03:07 | mriedem | i kid i kid | |
| 23:04:17 | mriedem | i think as long as you have HPC in there somewhere it will hit enough marketing bells to satisfy the entry | |
| 23:04:46 | mriedem | "Support virtual persistent memory devices for HPC workloads when using the libvirt driver.": | |
| 23:04:48 | mriedem | something like that | |
| 23:04:49 | efried | "edge" | |
| 23:04:51 | sean-k-mooney | the primary usecase is really something like "vPMEM support was intoduced to allow big data workloads to retain more data in persitent memroy reduceing the total cost of onership for big data clouds" | |
| 23:05:24 | sean-k-mooney | ya hpc is the other vertical that will likely use it most | |
| 23:05:30 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 23:05:31 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 23:05:37 | efried | mriedem, logan-: Howza ^ | |
| 23:05:46 | mriedem | "other vertical"? someone get this guy a c-suite office | |
| 23:06:00 | mriedem | efried: it's dinner o'clock for me | |
| 23:06:03 | efried | me too | |
| 23:06:26 | sean-k-mooney | its tuesday for me so im going to sleep soon o/ | |
| 23:07:35 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 23:07:36 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 23:11:19 | alex_xu | whatever that is worth to celebrate :) | |
| #openstack-nova - 2019-09-17 | |||
| 00:16:05 | alex_xu | efried: updated https://review.opendev.org/682509 | |
| 00:17:02 | zzzeek | logan-: not something ive seen before, https://dba.stackexchange.com/a/19139/81161 seems to have the best info | |
| 00:17:10 | zzzeek | max_packet_size | |
| 00:19:22 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511 | |
| 00:20:04 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511 | |
| 00:38:15 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511 | |
| 00:45:31 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add the maximum train flag in api version history https://review.opendev.org/682513 | |
| 01:41:37 | openstackgerrit | Merged openstack/nova master: Follow up for the bandwidth series https://review.opendev.org/682389 | |
| 01:41:44 | openstackgerrit | Merged openstack/nova master: Remove redundancies from AggregateRequestFiltersTest.setUp https://review.opendev.org/682475 | |
| 01:42:11 | openstackgerrit | Merged openstack/nova master: doc: mark the max microversion for train https://review.opendev.org/681982 | |
| 01:56:58 | openstackgerrit | hutianhao27 proposed openstack/nova master: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/682523 | |
| 02:44:57 | openstackgerrit | ya.wang proposed openstack/nova master: Fix typo of cpu model when check CPU compatibility https://review.opendev.org/682267 | |
| 05:16:56 | openstackgerrit | Merged openstack/nova master: Centralize volume create code during boot from volume https://review.opendev.org/682378 | |
| 06:20:23 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969 | |
| 07:14:00 | bauzas | good morning Nova | |
| 07:18:53 | gibi | good morning bauzas | |
| 07:43:22 | bauzas | gibi: any change you wanna stuff me in ? | |
| 07:59:37 | gibi | bauzas: nothing open at the moment I'm working on to fix mriedem's comments in the bug fix https://review.opendev.org/#/c/666857/ | |
| 07:59:52 | bauzas | ack, saw iy | |
| 07:59:54 | bauzas | it* | |
| 08:00:01 | gibi | bauzas: I will ping you if that rework is ready | |
| 08:01:00 | bauzas | ++ | |
| 08:01:56 | gibi | bauzas: do you have something I need to look at? | |
| 08:02:28 | bauzas | gibi: not atm, still fixing some tests for the placement audit | |
| 08:06:43 | openstackgerrit | Yongli He proposed openstack/nova master: clean up orphan instances https://review.opendev.org/627765 | |
| 08:06:43 | openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.opendev.org/648912 | |
| 08:23:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857 | |
| 09:41:12 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Remove stale BDMs on reserve_block_device_name failure https://review.opendev.org/682594 | |
| 11:15:52 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM libvirt: extract _start_heterogeneous_computes https://review.opendev.org/682620 | |
| 11:15:53 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Add regression test for bug #1843639 https://review.opendev.org/682621 | |
| 11:15:54 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/682622 | |
| 11:15:54 | openstack | bug 1843639 in OpenStack Compute (nova) "libvirt: post_live_migration failures to disconnect volumes result in the rollback of live migrations" [Medium,Confirmed] https://launchpad.net/bugs/1843639 | |
| 11:35:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857 | |
| 11:57:49 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: VMware VMDK detach: get adapter type from instance VM https://review.opendev.org/653738 | |
| 11:58:19 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/673790 | |
| 11:58:20 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Methods to handle request for Secure Boot & non-Q35 machine types https://review.opendev.org/682628 | |
| 11:58:20 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Add methods to detect firmware auto-selection and SB https://review.opendev.org/682627 | |
| 12:01:38 | gibi | mriedem, bauzas: comments fixed i https://review.opendev.org/#/c/666857 | |
| 12:05:41 | stephenfin | efried: Are we ever going to be able to merge a patch again? :D | |
| 12:10:46 | kashyap | (And remove needless compat code; we should have much less of it, though, as we've already cleaned up a good chunk of it a few months.) | |
| 12:17:25 | efried | stephenfin: appears this time it was mirror problems. But yeah, the gods really seem to hate that top pmem patch. | |
| 12:17:52 | efried | stephenfin: though mriedem and I fixed a regression from the forbidden aggs series which was causing a bunch of our failures. | |
| 12:18:11 | stephenfin | o rly? link? | |
| 12:18:12 | efried | now if we could just kill the innodb bug | |
| 12:18:28 | efried | stephenfin: https://review.opendev.org/#/c/682475/ | |
| 12:18:57 | efried | also a couple WIPs to poison multiple sched services from starting. | |
| 12:19:23 | sean-k-mooney | kashyap: didnt we already bump to a minium of 3.0 earlier in the cycle | |
| 12:19:28 | efried | https://review.opendev.org/#/c/682485/ and https://review.opendev.org/#/c/682486/ | |