| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-13 | |||
| 18:00:06 | mriedem | wtf is that | |
| 18:00:10 | artom | Picking up on the 5th and 6th | |
| 18:00:25 | mriedem | artom: we only have 10 days of logs | |
| 18:00:35 | artom | I thought it was 30? | |
| 18:00:44 | mriedem | 10 days of indexed logs in ES | |
| 18:00:50 | artom | Aha | |
| 18:01:02 | mriedem | and we need to filter out glance, | |
| 18:01:07 | mriedem | but yeah i can check that in logstash | |
| 18:01:42 | sean-k-mooney | i think htat is the empty buffer that subunit was trying to use or somthing like that | |
| 18:01:57 | sean-k-mooney | ValueError: Not a text type application/octet-stream | |
| 18:02:20 | sean-k-mooney | was that a local issue or was that just one of the things we saw when subunit blows up | |
| 18:02:55 | mriedem | you get the valuerror when subunit blows up | |
| 18:03:00 | mriedem | i have'nt seen that wall of 00s before though, | |
| 18:03:03 | mriedem | and in the unit test fail https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_897/682025/1/gate/openstack-tox-py37/89752ac/job-output.txt | |
| 18:03:11 | mriedem | dumping the libvirt host capabilities seems to be new | |
| 18:03:19 | mriedem | which i wouldn't be surprised if something from the sev series added that | |
| 18:03:32 | mriedem | INFO [nova.virt.libvirt.host] kernel doesn't support AMD SEV" | |
| 18:03:44 | sean-k-mooney | ya i saw that | |
| 18:04:28 | sean-k-mooney | that might be related although i did not see that in the other failed run | |
| 18:04:34 | sean-k-mooney | https://4c970d31e6335324d673-0d0b23b7f62d479822a1b52b605369e3.ssl.cf1.rackcdn.com/682025/1/gate/nova-tox-functional-py36/5f76307/testr_results.html.gz | |
| 18:04:39 | mriedem | the other failed run is a functional job | |
| 18:05:26 | mriedem | looks like nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_xml_and_uri ? | |
| 18:05:33 | sean-k-mooney | same wall of 00s | |
| 18:06:31 | sean-k-mooney | oh damb i got the same issue locally | |
| 18:06:34 | sean-k-mooney | with master | |
| 18:08:19 | mriedem | if you look at a functional-py36 run that didn't fail, you'll still see lots of stuff dumped to output on passing tests https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_529/656594/7/gate/nova-tox-functional-py36/529d847/job-output.txt | |
| 18:08:35 | mriedem | test_create_server_with_network_create_port_has_resource_request | |
| 18:08:39 | sean-k-mooney | like this http://paste.openstack.org/show/775781/ | |
| 18:08:43 | mriedem | test_boot_reschedule_fill_provider_mapping_raises | |
| 18:09:31 | artom | Dumb solution, but move that capabilities logging to debug? | |
| 18:09:31 | sean-k-mooney | its proably the 6 INFO [nova.virt.libvirt.host] Libvirt host capabilities <capabilities>' call | |
| 18:09:41 | mriedem | artom: i have a better solution | |
| 18:09:44 | mriedem | please hold | |
| 18:10:08 | mriedem | with anticipation? | |
| 18:10:15 | artom | Partially. | |
| 18:12:23 | sean-k-mooney | so we remove https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L660 yes | |
| 18:12:32 | sean-k-mooney | you mock it in the tests | |
| 18:12:42 | mriedem | i'm going to move it into a separate method and stub it out in the libvirt fixture | |
| 18:13:07 | sean-k-mooney | mriedem: ya that is a good way to handel that | |
| 18:13:47 | artom | What about tests that don't use the fixture? All the fakelibvirt'y tests | |
| 18:13:51 | sean-k-mooney | normally this would only print once as we cache the result | |
| 18:14:52 | artom | Or just move it to init_host | |
| 18:15:02 | artom | And not on every damn get_capabilities call | |
| 18:15:16 | sean-k-mooney | it does not log on every call | |
| 18:15:30 | sean-k-mooney | it logs once but we keep recreating the driver in the tests | |
| 18:15:34 | artom | Ah, right, only if self._caps isn't set | |
| 18:15:40 | sean-k-mooney | yes | |
| 18:15:54 | mriedem | LibvirtConnTestCase is the bad one and that uses the fixture so i think we'll be ok | |
| 18:21:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: stub logging of host capabilities https://review.opendev.org/682133 | |
| 18:21:33 | mriedem | tremble before this ^ | |
| 18:22:55 | sean-k-mooney | isnt it awsome how it conflits with everything | |
| 18:23:18 | mriedem | ha | |
| 18:23:23 | sean-k-mooney | im running the tests localy | |
| 18:23:25 | artom | Don't we have a thing about not changing code to work around CI? | |
| 18:23:37 | sean-k-mooney | this is not a ci bug | |
| 18:23:45 | sean-k-mooney | its a subunit parser bug | |
| 18:23:49 | artom | Tru | |
| 18:23:50 | artom | e | |
| 18:24:31 | sean-k-mooney | so the alteritive is disbable test. which is worse or 1 day after FF replace our test framework with somethin that does not use stestr | |
| 18:24:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22subunit.parser%5C%22%20AND%20message%3A%5C%22FAILED%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=7d | |
| 18:24:38 | mriedem | for the recent nova hits on that bug | |
| 18:24:48 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22subunit.parser%5C%22%20AND%20message%3A%5C%22FAILED%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1%20AND%20project%3A%5C%22openstack%2Fnova%5C%22&from=10d i guess is better | |
| 18:27:35 | sean-k-mooney | so far it seams to be working locallly. do you want to rebases https://review.opendev.org/#/c/682025/ on top of combine them? | |
| 18:27:52 | mriedem | i do'nt want to combine them | |
| 18:28:00 | mriedem | and i don't know if we have cores around to approve the libvirt one | |
| 18:28:40 | mriedem | looking at the other tracebacks in a passing functional test job, i see a lot of _allocate_network_async which makes me think (and i think these are negative scenario tests) that we're triggering a failure to satisfy the test, | |
| 18:28:47 | mriedem | but then dropping something in the code, | |
| 18:29:03 | mriedem | which gets logged and i'm thinking we might want to have a fixture that makes _allocate_network_async serial | |
| 18:29:12 | mriedem | like the fixtures we have for spawn_n | |
| 18:29:51 | mriedem | i mean we're hitting NetworkInfoAsyncWrapper which calls utils.spawn(_allocate_network_async) | |
| 18:29:53 | sean-k-mooney | we use a fixture like that for the cells scater gather code too right | |
| 18:30:05 | mriedem | so we might just not be using the spawn fixture in several places | |
| 18:33:16 | sean-k-mooney | there are quite a few SADeprecationWarning too | |
| 18:33:58 | sean-k-mooney | The Engine.contextual_connect() method is deprecated. | |
| 18:34:07 | mriedem | and lots of "Detected removed column" | |
| 18:34:29 | mriedem | we should probably squash alembic logging in tests | |
| 18:35:25 | sean-k-mooney | well im not sure we would notice it was deprected if we did but maybe | |
| 18:35:41 | mriedem | i'm talking about a different issue | |
| 18:35:47 | mriedem | these | |
| 18:35:47 | mriedem | 2019-09-13 17:33:40.316170 | ubuntu-bionic | 2019-09-13 17:33:40.248 4439 INFO alembic.autogenerate.compare [req-d6f4db5b-4ad9-4096-9832-08d8692555d7 fake fake - - -] Detected removed column 'build_requests.vm_state'[00m | |
| 18:36:10 | sean-k-mooney | oh ok | |
| 18:36:30 | mriedem | there are columns from the build_requests table that were removed from the model but not the schema | |
| 18:37:04 | sean-k-mooney | like the vpmem coloum in the instance_extra table | |
| 18:38:07 | sean-k-mooney | dont we normally whitelist to be ingored. i would have expected that to not be logged | |
| 18:38:10 | sean-k-mooney | but i guess not | |
| 18:38:26 | mriedem | idk | |
| 18:38:35 | mriedem | idk where the message is coming from | |
| 18:38:42 | mriedem | probably the walk migrations stuff | |
| 18:39:26 | mriedem | yeah there is a whitelist in filter_metadata_diff | |
| 18:39:56 | sean-k-mooney | the deprecation warnings at lesast are not coming form nova they are form sqlaclamate migrate | |
| 18:40:06 | sean-k-mooney | eventrulally we will need to move off that | |
| 18:40:08 | mriedem | yeah that's an old one | |
| 18:40:16 | mriedem | i disagree | |
| 18:40:55 | mriedem | just fix this https://bugs.launchpad.net/sqlalchemy-migrate/+bug/1814288 | |
| 18:40:56 | openstack | Launchpad bug 1814288 in sqlalchemy-migrate "DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead" [Medium,Confirmed] | |
| 18:41:25 | mriedem | sqlalchemy.migrate is opened by openstack, i'm a core, it's in maintenance mode, and changing nova to go from migrate to alembic is not worth the trouble | |
| 18:41:26 | sean-k-mooney | mriedem: didnt the maintainer say the did not want to maintain sqlalcamey migrate anymore | |
| 18:41:28 | mriedem | at least not while i'm around | |
| 18:41:36 | sean-k-mooney | oh ok | |
| 18:41:37 | mriedem | openstack has maintained it for like 5 years now | |