| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-13 | |||
| 16:41:03 | sean-k-mooney | wow that was the fix... | |
| 16:44:18 | stephenfin | yup | |
| 16:44:32 | stephenfin | tbf, I've made that same mistake myself many times | |
| 16:44:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: DNM: Reproducer for bug #1843836 https://review.opendev.org/682111 | |
| 16:44:41 | openstack | bug 1843836 in OpenStack Compute (nova) "Failure to schedule if flavor contains non-CPU flag traits" [Undecided,In progress] https://launchpad.net/bugs/1843836 - Assigned to Stephen Finucane (stephenfinucane) | |
| 16:44:43 | stephenfin | mriedem_afk: ^ | |
| 16:44:58 | sean-k-mooney | i think it was enqued directly to gate so it should merge soon i hope | |
| 16:45:43 | stephenfin | mriedem_afk: there's a WIP functional reproducer for the bug. I'll clean it up Monday once I'm done investigating the other issue that reproducer highlighted | |
| 16:45:50 | stephenfin | *potential issue | |
| 16:45:55 | stephenfin | sean-k-mooney: It sure was, yup | |
| 16:47:49 | aspiers | stephenfin: that github link is basically a DoS attack on my browser X-D | |
| 16:48:03 | stephenfin | aspiers: Hahahaha | |
| 16:48:17 | stephenfin | yeah...don't try to load it on a phone | |
| 16:48:49 | aspiers | no that was on my laptop | |
| 16:49:08 | stephenfin | if there was ever a sign that a file had grown too large... | |
| 16:49:15 | aspiers | right? | |
| 16:49:30 | stephenfin | funnily enough, libvirt source files tend to be huuuuge as well | |
| 16:49:57 | sean-k-mooney | thats not uncommon in c | |
| 17:00:15 | aspiers | stephenfin: anyway yes it looks wrong to me | |
| 17:01:22 | stephenfin | \o/ | |
| 17:01:32 | stephenfin | Another one to fix on Monday | |
| 17:01:42 | stephenfin | Thanks for taking a look, aspiers | |
| 17:17:55 | aspiers | BTW I'm off next week | |
| 17:18:11 | aspiers | stephenfin: In case you missed it, I've just drafted a blog post on SEV. Feedback welcome: https://blog.adamspiers.org/?p=1871&preview=1&_ppp=1bfc4e4f7e | |
| 17:23:37 | stephenfin | aspiers: Won't get to it today but I'll try early next week. Enjoy your time off! | |
| 17:25:59 | aspiers | cheers :) | |
| 17:28:23 | artom | Where's the part you call us all poopyheads for not reviewing your code enough? | |
| 17:49:13 | sean-k-mooney | damit... mriedem_afk https://review.opendev.org/#/c/682025/ is going to fail | |
| 17:50:34 | sean-k-mooney | subunit parser is exploding | |
| 17:51:33 | mriedem | drats | |
| 17:51:55 | sean-k-mooney | https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_897/682025/1/gate/openstack-tox-py37/89752ac/testr_results.html.gz | |
| 17:53:02 | artom | What even causes those? | |
| 17:53:32 | sean-k-mooney | large amount of output to standard out or error | |
| 17:53:59 | sean-k-mooney | like all the logging of the host capablities | |
| 17:54:48 | mriedem | those seem to have spiked against recently so something must have merged that is causing us to tip over | |
| 17:54:57 | mriedem | http://status.openstack.org/elastic-recheck/#1813147 | |
| 17:55:07 | mriedem | #2 top most fail in the check queue right now | |
| 17:55:49 | sean-k-mooney | its happening in other jobs too not just py35 but ya | |
| 17:56:13 | sean-k-mooney | before it was due to a bunch of deprecation warnings | |
| 17:57:35 | mriedem | the name of the bug is old | |
| 17:58:18 | sean-k-mooney | can we force submit that change? im running the unit test locally to see if it explodes but i dont expect it to | |
| 17:58:37 | mriedem | as in get it re-queued? | |
| 17:58:44 | mriedem | i think you'd have to rebase | |
| 17:58:50 | mriedem | which will put it back through check | |
| 17:58:59 | sean-k-mooney | as in ask infra to override zuul | |
| 17:59:00 | mriedem | i asked for it to be promoted to gate earlier in infra | |
| 17:59:08 | sean-k-mooney | but we could requeu it too | |
| 17:59:45 | mriedem | jesus look at this https://4c970d31e6335324d673-0d0b23b7f62d479822a1b52b605369e3.ssl.cf1.rackcdn.com/682025/1/gate/nova-tox-functional-py36/5f76307/job-output.txt | |
| 17:59:49 | mriedem | search for inprogress | |
| 17:59:51 | mriedem | see the wall of 00s | |
| 18:00:02 | artom | Those subunit things appear to start on september... 4th? | |
| 18:00:06 | mriedem | wtf is that | |
| 18:00:10 | artom | Picking up on the 5th and 6th | |
| 18:00:25 | mriedem | artom: we only have 10 days of logs | |
| 18:00:35 | artom | I thought it was 30? | |
| 18:00:44 | mriedem | 10 days of indexed logs in ES | |
| 18:00:50 | artom | Aha | |
| 18:01:02 | mriedem | and we need to filter out glance, | |
| 18:01:07 | mriedem | but yeah i can check that in logstash | |
| 18:01:42 | sean-k-mooney | i think htat is the empty buffer that subunit was trying to use or somthing like that | |
| 18:01:57 | sean-k-mooney | ValueError: Not a text type application/octet-stream | |
| 18:02:20 | sean-k-mooney | was that a local issue or was that just one of the things we saw when subunit blows up | |
| 18:02:55 | mriedem | you get the valuerror when subunit blows up | |
| 18:03:00 | mriedem | i have'nt seen that wall of 00s before though, | |
| 18:03:03 | mriedem | and in the unit test fail https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_897/682025/1/gate/openstack-tox-py37/89752ac/job-output.txt | |
| 18:03:11 | mriedem | dumping the libvirt host capabilities seems to be new | |
| 18:03:19 | mriedem | which i wouldn't be surprised if something from the sev series added that | |
| 18:03:32 | mriedem | INFO [nova.virt.libvirt.host] kernel doesn't support AMD SEV" | |
| 18:03:44 | sean-k-mooney | ya i saw that | |
| 18:04:28 | sean-k-mooney | that might be related although i did not see that in the other failed run | |
| 18:04:34 | sean-k-mooney | https://4c970d31e6335324d673-0d0b23b7f62d479822a1b52b605369e3.ssl.cf1.rackcdn.com/682025/1/gate/nova-tox-functional-py36/5f76307/testr_results.html.gz | |
| 18:04:39 | mriedem | the other failed run is a functional job | |
| 18:05:26 | mriedem | looks like nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_xml_and_uri ? | |
| 18:05:33 | sean-k-mooney | same wall of 00s | |
| 18:06:31 | sean-k-mooney | oh damb i got the same issue locally | |
| 18:06:34 | sean-k-mooney | with master | |
| 18:08:19 | mriedem | if you look at a functional-py36 run that didn't fail, you'll still see lots of stuff dumped to output on passing tests https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_529/656594/7/gate/nova-tox-functional-py36/529d847/job-output.txt | |
| 18:08:35 | mriedem | test_create_server_with_network_create_port_has_resource_request | |
| 18:08:39 | sean-k-mooney | like this http://paste.openstack.org/show/775781/ | |
| 18:08:43 | mriedem | test_boot_reschedule_fill_provider_mapping_raises | |
| 18:09:31 | artom | Dumb solution, but move that capabilities logging to debug? | |
| 18:09:31 | sean-k-mooney | its proably the 6 INFO [nova.virt.libvirt.host] Libvirt host capabilities <capabilities>' call | |
| 18:09:41 | mriedem | artom: i have a better solution | |
| 18:09:44 | mriedem | please hold | |
| 18:10:08 | mriedem | with anticipation? | |
| 18:10:15 | artom | Partially. | |
| 18:12:23 | sean-k-mooney | so we remove https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L660 yes | |
| 18:12:32 | sean-k-mooney | you mock it in the tests | |
| 18:12:42 | mriedem | i'm going to move it into a separate method and stub it out in the libvirt fixture | |
| 18:13:07 | sean-k-mooney | mriedem: ya that is a good way to handel that | |
| 18:13:47 | artom | What about tests that don't use the fixture? All the fakelibvirt'y tests | |
| 18:13:51 | sean-k-mooney | normally this would only print once as we cache the result | |
| 18:14:52 | artom | Or just move it to init_host | |
| 18:15:02 | artom | And not on every damn get_capabilities call | |
| 18:15:16 | sean-k-mooney | it does not log on every call | |
| 18:15:30 | sean-k-mooney | it logs once but we keep recreating the driver in the tests | |
| 18:15:34 | artom | Ah, right, only if self._caps isn't set | |
| 18:15:40 | sean-k-mooney | yes | |
| 18:15:54 | mriedem | LibvirtConnTestCase is the bad one and that uses the fixture so i think we'll be ok | |
| 18:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: stub logging of host capabilities https://review.opendev.org/682133 | |