| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-16 | |||
| 22:27:14 | mriedem | there is probably some way to find average job time per node provider and nova openstack-tox-py36 job or something | |
| 22:27:16 | sean-k-mooney | infra have many grapha dashborads that track things but i dont know if they yave abuild time per regoin one | |
| 22:27:25 | sean-k-mooney | you could proably figure it out form logstash | |
| 22:27:42 | mriedem | i think awhile back i had wondered if those nodes used some different mysql binary or something | |
| 22:28:16 | mriedem | for everything to be not crazy though it should be the same ubuntu 18.04 image and such regardless of node provider | |
| 22:28:23 | sean-k-mooney | they should all use the nodepool image and then hit the infra mirros which are avaible via afs in all regions right | |
| 22:28:26 | sean-k-mooney | so it shoudl be the same | |
| 22:29:17 | efried | http://zuul.openstack.org/builds?job_name=openstack-tox-py27&branch=master&project=openstack%2Fnova but this table doesn't show the node provider... | |
| 22:29:21 | sean-k-mooney | i think infra has been uploading its own image nightly since the zuul v2 days to not have random things break in diffrent clouds | |
| 22:30:16 | logan- | o/ from limestone | |
| 22:30:50 | efried | this slow bastard happened on rax https://zuul.opendev.org/t/openstack/build/38181c3314284545943b38b52c72f8d7/log/job-output.txt | |
| 22:31:54 | sean-k-mooney | logan- they are trying to debug a test that seam to be hitting a timeout | |
| 22:32:16 | sean-k-mooney | logan-: but only (mostly?) happens in limestone-regionone | |
| 22:33:03 | alex_xu | efried: I'm here | |
| 22:33:04 | sean-k-mooney | logan-: so we were wondering if limesotne is generally slower the other providre and hiting the timeout more often or if it could be a config/could issue | |
| 22:33:38 | sean-k-mooney | logan-: e.g. is there something differnet that could make it fail more often on limestone or it that just a red herring | |
| 22:33:45 | efried | alex_xu: Can you, or Rui, or luyao, or someone please formulate a bullet for vpmem for the train cycle highlights in a followon to https://review.opendev.org/#/c/681943/ ? | |
| 22:34:11 | alex_xu | efried: got it, will do today | |
| 22:34:25 | efried | alex_xu: If you seed the message, I'll scrub the grammar etc. Thanks. | |
| 22:34:38 | alex_xu | efried: thanks | |
| 22:35:38 | logan- | i'm not sure if our testcloud is slower than other nodepool providers. certainly the image would be the same as any other nodepool node. looking thru the log.. i'm curious if the job hangs somewhere specific, or if its just slower in general. and if our nodes are just showing up slower on average i'd like to know so we can figure out if it is h/w or config related. | |
| 22:36:23 | efried | logan-: The thing we're trying to nail down is http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22sqlalchemy.exc.InterfaceError:%20(pymysql.err.InterfaceError)%20(0,%20'')%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s | |
| 22:36:48 | efried | we consistently see the test suite time out on the same 1-3 tests | |
| 22:39:22 | efried | logan-: and if you punch the `node_provider` checkbox on the left-hand side next to the result list, you see it's *always* limestone-regionone | |
| 22:40:52 | logan- | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/mysql/error.log are any of these messages at the bottom pertinent? | |
| 22:42:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.opendev.org/541420 | |
| 22:42:58 | mriedem | logan-: we're not sure, we need zzzeek's help for that | |
| 22:43:58 | mriedem | those are definitely related to the opportunistic db tests that fail, | |
| 22:44:07 | mriedem | the db is a random db name created by the test fixture, | |
| 22:44:12 | mriedem | openstack_citest is setup by a script on the node | |
| 22:44:32 | mriedem | https://github.com/openstack/nova/blob/master/tools/test-setup.sh does the mysql/postgresql setup | |
| 22:45:05 | efried | logan-: this might be what you were already looking at, but I did a compare of the mysql error log on a "good" run and a "bad" one, and didn't see significant differences. | |
| 22:45:21 | efried | logan-: see bottom-most comment on https://review.opendev.org/#/c/678051/ | |
| 22:45:57 | logan- | yep, that's what i was wondering, thanks | |
| 22:47:05 | efried | although the bad one is taking way longer... | |
| 22:47:12 | efried | but dunno if that's cause or effect | |
| 22:47:39 | logan- | to answer the general question -- I don't know of anything significantly different on our nodes. they're pretty standard dual proc intel systems with ssds. I wonder if there's something in particular with this test that is wrecking the node somehow. | |
| 22:48:49 | efried | mhm, mriedem here's an interesting data point: in the "good" one, all those "abort" messages come through in under a minute. In the "bad" one, the first 16 or so are under a minute, then we start getting long delays. | |
| 22:49:32 | efried | now, IIUC, everything before those "abort" messages is just setup, and the "abort" things are happening during the actual meat of the test. So I guess that's not really surprising. | |
| 22:50:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.opendev.org/541420 | |
| 22:53:37 | efried | It would be painful, but maybe if we knew what operations were being run, it would give us a clue as to what's happening when it starts to grind down. | |
| 22:54:08 | efried | anybody know how to turn on like trace logs to see the sql statements themselves as they're issued? | |
| 22:54:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Stop using volume_api.initialize_connection https://review.opendev.org/682508 | |
| 22:55:20 | mriedem | efried: i want to say https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or | |
| 22:55:25 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace | |
| 22:57:54 | alex_xu | efried: https://review.opendev.org/682509 | |
| 22:58:19 | efried | alex_xu: cool, thank you. | |
| 22:58:26 | alex_xu | efried: np! | |
| 22:58:36 | efried | alex_xu: since this is market-y, we probably don't want to mention the generic resource framework | |
| 22:58:49 | alex_xu | efried: ah, got it | |
| 22:58:59 | efried | that's a thing only devs would care about, at least until it reaps perf benefits via placement, which is a ways off. | |
| 22:59:14 | efried | alex_xu: perhaps kill that sentence and instead explain a bit more what vpmem is | |
| 22:59:15 | alex_xu | efried: sev mentioned that, that is why I follow | |
| 22:59:24 | efried | oh? looking... | |
| 23:00:05 | efried | their workloads when using the libvirt compute driver" | |
| 23:00:05 | efried | alex_xu: yeah, I'm looking for something corresponding to "to protect users against attackers or rogue administrators snooping on | |
| 23:00:16 | alex_xu | efried: got it | |
| 23:00:31 | alex_xu | mriedem: you are super fast | |
| 23:00:54 | efried | So like "Added $feature for $benefit" | |
| 23:01:56 | mriedem | "Added VPMEMs for Intel to sell hardware." | |
| 23:01:59 | mriedem | did it for you :) | |
| 23:02:29 | sean-k-mooney | add vpmem so the nsa can keep allyour data in "ram" to track you faster | |
| 23:02:41 | mriedem | "added vpmems so alex can continue working upstream" | |
| 23:03:07 | mriedem | i kid i kid | |
| 23:04:17 | mriedem | i think as long as you have HPC in there somewhere it will hit enough marketing bells to satisfy the entry | |
| 23:04:46 | mriedem | "Support virtual persistent memory devices for HPC workloads when using the libvirt driver.": | |
| 23:04:48 | mriedem | something like that | |
| 23:04:49 | efried | "edge" | |
| 23:04:51 | sean-k-mooney | the primary usecase is really something like "vPMEM support was intoduced to allow big data workloads to retain more data in persitent memroy reduceing the total cost of onership for big data clouds" | |
| 23:05:24 | sean-k-mooney | ya hpc is the other vertical that will likely use it most | |
| 23:05:30 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 23:05:31 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 23:05:37 | efried | mriedem, logan-: Howza ^ | |
| 23:05:46 | mriedem | "other vertical"? someone get this guy a c-suite office | |
| 23:06:00 | mriedem | efried: it's dinner o'clock for me | |
| 23:06:03 | efried | me too | |
| 23:06:26 | sean-k-mooney | its tuesday for me so im going to sleep soon o/ | |
| 23:07:35 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 23:07:36 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 23:11:19 | alex_xu | whatever that is worth to celebrate :) | |
| #openstack-nova - 2019-09-17 | |||
| 00:16:05 | alex_xu | efried: updated https://review.opendev.org/682509 | |
| 00:17:02 | zzzeek | logan-: not something ive seen before, https://dba.stackexchange.com/a/19139/81161 seems to have the best info | |
| 00:17:10 | zzzeek | max_packet_size | |
| 00:19:22 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511 | |
| 00:20:04 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511 | |
| 00:38:15 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Track user_id/project_id for migrations https://review.opendev.org/682511 | |
| 00:45:31 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add the maximum train flag in api version history https://review.opendev.org/682513 | |
| 01:41:37 | openstackgerrit | Merged openstack/nova master: Follow up for the bandwidth series https://review.opendev.org/682389 | |
| 01:41:44 | openstackgerrit | Merged openstack/nova master: Remove redundancies from AggregateRequestFiltersTest.setUp https://review.opendev.org/682475 | |
| 01:42:11 | openstackgerrit | Merged openstack/nova master: doc: mark the max microversion for train https://review.opendev.org/681982 | |
| 01:56:58 | openstackgerrit | hutianhao27 proposed openstack/nova master: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/682523 | |
| 02:44:57 | openstackgerrit | ya.wang proposed openstack/nova master: Fix typo of cpu model when check CPU compatibility https://review.opendev.org/682267 | |
| 05:16:56 | openstackgerrit | Merged openstack/nova master: Centralize volume create code during boot from volume https://review.opendev.org/682378 | |
| 06:20:23 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969 | |
| 07:14:00 | bauzas | good morning Nova | |
| 07:18:53 | gibi | good morning bauzas | |
| 07:43:22 | bauzas | gibi: any change you wanna stuff me in ? | |
| 07:59:37 | gibi | bauzas: nothing open at the moment I'm working on to fix mriedem's comments in the bug fix https://review.opendev.org/#/c/666857/ | |
| 07:59:52 | bauzas | ack, saw iy | |
| 07:59:54 | bauzas | it* | |