| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-13 | |||
| 16:49:08 | stephenfin | if there was ever a sign that a file had grown too large... | |
| 16:49:15 | aspiers | right? | |
| 16:49:30 | stephenfin | funnily enough, libvirt source files tend to be huuuuge as well | |
| 16:49:57 | sean-k-mooney | thats not uncommon in c | |
| 17:00:15 | aspiers | stephenfin: anyway yes it looks wrong to me | |
| 17:01:22 | stephenfin | \o/ | |
| 17:01:32 | stephenfin | Another one to fix on Monday | |
| 17:01:42 | stephenfin | Thanks for taking a look, aspiers | |
| 17:17:55 | aspiers | BTW I'm off next week | |
| 17:18:11 | aspiers | stephenfin: In case you missed it, I've just drafted a blog post on SEV. Feedback welcome: https://blog.adamspiers.org/?p=1871&preview=1&_ppp=1bfc4e4f7e | |
| 17:23:37 | stephenfin | aspiers: Won't get to it today but I'll try early next week. Enjoy your time off! | |
| 17:25:59 | aspiers | cheers :) | |
| 17:28:23 | artom | Where's the part you call us all poopyheads for not reviewing your code enough? | |
| 17:49:13 | sean-k-mooney | damit... mriedem_afk https://review.opendev.org/#/c/682025/ is going to fail | |
| 17:50:34 | sean-k-mooney | subunit parser is exploding | |
| 17:51:33 | mriedem | drats | |
| 17:51:55 | sean-k-mooney | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_897/682025/1/gate/openstack-tox-py37/89752ac/testr_results.html.gz | |
| 17:53:02 | artom | What even causes those? | |
| 17:53:32 | sean-k-mooney | large amount of output to standard out or error | |
| 17:53:59 | sean-k-mooney | like all the logging of the host capablities | |
| 17:54:48 | mriedem | those seem to have spiked against recently so something must have merged that is causing us to tip over | |
| 17:54:57 | mriedem | http://status.openstack.org/elastic-recheck/#1813147 | |
| 17:55:07 | mriedem | #2 top most fail in the check queue right now | |
| 17:55:49 | sean-k-mooney | its happening in other jobs too not just py35 but ya | |
| 17:56:13 | sean-k-mooney | before it was due to a bunch of deprecation warnings | |
| 17:57:35 | mriedem | the name of the bug is old | |
| 17:58:18 | sean-k-mooney | can we force submit that change? im running the unit test locally to see if it explodes but i dont expect it to | |
| 17:58:37 | mriedem | as in get it re-queued? | |
| 17:58:44 | mriedem | i think you'd have to rebase | |
| 17:58:50 | mriedem | which will put it back through check | |
| 17:58:59 | sean-k-mooney | as in ask infra to override zuul | |
| 17:59:00 | mriedem | i asked for it to be promoted to gate earlier in infra | |
| 17:59:08 | sean-k-mooney | but we could requeu it too | |
| 17:59:45 | mriedem | jesus look at this https://4c970d31e6335324d673-0d0b23b7f62d479822a1b52b605369e3.ssl.cf1.rackcdn.com/682025/1/gate/nova-tox-functional-py36/5f76307/job-output.txt | |
| 17:59:49 | mriedem | search for inprogress | |
| 17:59:51 | mriedem | see the wall of 00s | |
| 18:00:02 | artom | Those subunit things appear to start on september... 4th? | |
| 18:00:06 | mriedem | wtf is that | |
| 18:00:10 | artom | Picking up on the 5th and 6th | |
| 18:00:25 | mriedem | artom: we only have 10 days of logs | |
| 18:00:35 | artom | I thought it was 30? | |
| 18:00:44 | mriedem | 10 days of indexed logs in ES | |
| 18:00:50 | artom | Aha | |
| 18:01:02 | mriedem | and we need to filter out glance, | |
| 18:01:07 | mriedem | but yeah i can check that in logstash | |
| 18:01:42 | sean-k-mooney | i think htat is the empty buffer that subunit was trying to use or somthing like that | |
| 18:01:57 | sean-k-mooney | ValueError: Not a text type application/octet-stream | |
| 18:02:20 | sean-k-mooney | was that a local issue or was that just one of the things we saw when subunit blows up | |
| 18:02:55 | mriedem | you get the valuerror when subunit blows up | |
| 18:03:00 | mriedem | i have'nt seen that wall of 00s before though, | |
| 18:03:03 | mriedem | and in the unit test fail https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_897/682025/1/gate/openstack-tox-py37/89752ac/job-output.txt | |
| 18:03:11 | mriedem | dumping the libvirt host capabilities seems to be new | |
| 18:03:19 | mriedem | which i wouldn't be surprised if something from the sev series added that | |
| 18:03:32 | mriedem | INFO [nova.virt.libvirt.host] kernel doesn't support AMD SEV" | |
| 18:03:44 | sean-k-mooney | ya i saw that | |
| 18:04:28 | sean-k-mooney | that might be related although i did not see that in the other failed run | |
| 18:04:34 | sean-k-mooney | https://4c970d31e6335324d673-0d0b23b7f62d479822a1b52b605369e3.ssl.cf1.rackcdn.com/682025/1/gate/nova-tox-functional-py36/5f76307/testr_results.html.gz | |
| 18:04:39 | mriedem | the other failed run is a functional job | |
| 18:05:26 | mriedem | looks like nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_xml_and_uri ? | |
| 18:05:33 | sean-k-mooney | same wall of 00s | |
| 18:06:31 | sean-k-mooney | oh damb i got the same issue locally | |
| 18:06:34 | sean-k-mooney | with master | |
| 18:08:19 | mriedem | if you look at a functional-py36 run that didn't fail, you'll still see lots of stuff dumped to output on passing tests https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_529/656594/7/gate/nova-tox-functional-py36/529d847/job-output.txt | |
| 18:08:35 | mriedem | test_create_server_with_network_create_port_has_resource_request | |
| 18:08:39 | sean-k-mooney | like this http://paste.openstack.org/show/775781/ | |
| 18:08:43 | mriedem | test_boot_reschedule_fill_provider_mapping_raises | |
| 18:09:31 | sean-k-mooney | its proably the 6 INFO [nova.virt.libvirt.host] Libvirt host capabilities <capabilities>' call | |
| 18:09:31 | artom | Dumb solution, but move that capabilities logging to debug? | |
| 18:09:41 | mriedem | artom: i have a better solution | |
| 18:09:44 | mriedem | please hold | |
| 18:10:08 | mriedem | with anticipation? | |
| 18:10:15 | artom | Partially. | |
| 18:12:23 | sean-k-mooney | so we remove https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L660 yes | |
| 18:12:32 | sean-k-mooney | you mock it in the tests | |
| 18:12:42 | mriedem | i'm going to move it into a separate method and stub it out in the libvirt fixture | |
| 18:13:07 | sean-k-mooney | mriedem: ya that is a good way to handel that | |
| 18:13:47 | artom | What about tests that don't use the fixture? All the fakelibvirt'y tests | |
| 18:13:51 | sean-k-mooney | normally this would only print once as we cache the result | |
| 18:14:52 | artom | Or just move it to init_host | |
| 18:15:02 | artom | And not on every damn get_capabilities call | |
| 18:15:16 | sean-k-mooney | it does not log on every call | |
| 18:15:30 | sean-k-mooney | it logs once but we keep recreating the driver in the tests | |
| 18:15:34 | artom | Ah, right, only if self._caps isn't set | |
| 18:15:40 | sean-k-mooney | yes | |
| 18:15:54 | mriedem | LibvirtConnTestCase is the bad one and that uses the fixture so i think we'll be ok | |
| 18:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: stub logging of host capabilities https://review.opendev.org/682133 | |
| 18:21:33 | mriedem | tremble before this ^ | |
| 18:22:55 | sean-k-mooney | isnt it awsome how it conflits with everything | |
| 18:23:18 | mriedem | ha | |
| 18:23:23 | sean-k-mooney | im running the tests localy | |
| 18:23:25 | artom | Don't we have a thing about not changing code to work around CI? | |
| 18:23:37 | sean-k-mooney | this is not a ci bug | |
| 18:23:45 | sean-k-mooney | its a subunit parser bug | |
| 18:23:49 | artom | Tru | |
| 18:23:50 | artom | e | |
| 18:24:31 | sean-k-mooney | so the alteritive is disbable test. which is worse or 1 day after FF replace our test framework with somethin that does not use stestr | |
| 18:24:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22subunit.parser%5C%22%20AND%20message%3A%5C%22FAILED%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d | |
| 18:24:38 | mriedem | for the recent nova hits on that bug | |
| 18:24:48 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22subunit.parser%5C%22%20AND%20message%3A%5C%22FAILED%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=10d i guess is better | |
| 18:27:35 | sean-k-mooney | so far it seams to be working locallly. do you want to rebases https://review.opendev.org/#/c/682025/ on top of combine them? | |