| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-17 | |||
| 09:37:21 | eandersson | http://paste.openstack.org/show/EUjnd5TWftogxGIVaSNP/ | |
| 09:37:25 | eandersson | This is the last log line | |
| 09:37:35 | eandersson | from the conductor / scheduler | |
| 09:38:13 | eandersson | yea bauzas nothing interesting in the events | |
| 09:39:06 | eandersson | Everything looks to be scheduled fine | |
| 09:39:21 | sean-k-mooney | right that is the last log for the instace we expect to see in the conductor log | |
| 09:39:54 | sean-k-mooney | well unless there is an error | |
| 09:40:00 | eandersson | no errors in the logs | |
| 09:40:08 | eandersson | :'( | |
| 09:40:36 | sean-k-mooney | and nothing in the compute agent long in debug either | |
| 09:40:41 | sean-k-mooney | it just disappars? | |
| 09:40:48 | eandersson | I pasted some logs early on | |
| 09:41:00 | sean-k-mooney | you captured the rpc right beign recived | |
| 09:41:03 | eandersson | let me check again | |
| 09:41:14 | sean-k-mooney | oh ill scroll back | |
| 09:45:26 | tssurya | eandersson: what's the exact boot request you are making ? | |
| 09:45:47 | sean-k-mooney | i dont see the logs for the compute scrolling back unfortunetly. | |
| 09:46:08 | eandersson | sean-k-mooney, I think the logs we discussed were things like | |
| 09:46:14 | eandersson | > instance x has been scheduled to this compute host | |
| 09:47:02 | eandersson | Let me enable even more debug logs | |
| 09:47:57 | sean-k-mooney | ah am you can re disable debug on the conductor/schduler since we are pretty sure its getting to the compute node at this point right | |
| 09:48:14 | eandersson | Yea already disabled | |
| 09:48:18 | eandersson | generates a lot of logs :p | |
| 09:48:59 | sean-k-mooney | for 200 servers with all servceince in debug i belive its something like 30GB a day in uncompressed logs | |
| 09:49:23 | sean-k-mooney | thats what kolla-ansibel said in there configuration docs in anycase | |
| 09:49:42 | eandersson | hehe well that is without kubernetes, octavia etc hitting your apis constantly | |
| 09:49:50 | eandersson | plus all other automation | |
| 09:50:19 | sean-k-mooney | right you would really want log rotate to be working well if you use debug always | |
| 09:51:17 | sean-k-mooney | we shoudl try to have more of a blance however. e.g. make info more useful without being just spam | |
| 09:52:06 | eandersson | On compute I see a message being received, but nothing more | |
| 09:52:29 | eandersson | > DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: ad3b7ccda9944e9e87f497f4401aa0b4 __call__ | |
| 09:52:57 | sean-k-mooney | ok so its not building the libvirt xml or plugging any ports | |
| 09:53:09 | sean-k-mooney | its just reciving it | |
| 09:53:14 | eandersson | yea | |
| 09:53:28 | sean-k-mooney | did you also check with the instance uuid | |
| 09:53:34 | eandersson | I bet this compute is bust, but everything seems fine (including already existing vms) | |
| 09:53:39 | eandersson | yea | |
| 09:53:59 | eandersson | I do get some logs with the instance id | |
| 09:54:06 | eandersson | but it's just the ones I posted earlier | |
| 09:54:20 | sean-k-mooney | ok | |
| 09:54:22 | eandersson | > Instance ... has been scheduled to this compute host, the scheduler has made an allocation against this compute node but the instance has yet to start. Skipping heal of allocation: ... | |
| 09:54:46 | sean-k-mooney | right that is the periodic task | |
| 09:55:04 | sean-k-mooney | that make sure the resouce useage on the host matches what we expect | |
| 09:55:44 | sean-k-mooney | that is normal to see between when the host has been selected in the db as the target for the instnace and the vm starting | |
| 09:57:34 | eandersson | Anyway bed time. It's 3AM. | |
| 09:57:51 | eandersson | Thanks for helping sean-k-mooney and gibi | |
| 09:58:08 | sean-k-mooney | eandersson: o/ get some rest | |
| 10:32:24 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| 10:33:31 | openstackgerrit | Merged openstack/nova stable/queens: Fix 'has_calls' method calls in unit tests https://review.opendev.org/677378 | |
| 10:42:09 | gibi | it is only me or the powervm unit tests started failing on master? | |
| 10:44:29 | gibi | AttributeError: 'DiGraph' object has no attribute 'node' | |
| 10:47:55 | gibi | https://702b7e8f253d29e679a6-2fe3f6c342189909aad5220492fb4721.ssl.cf1.rackcdn.com/688387/5/check/openstack-tox-cover/2c6410c/testr_results.html.gz | |
| 10:48:51 | gibi | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttributeError%3A%20'DiGraph'%20object%20has%20no%20attribute%20'node'%5C%22&from=7d | |
| 10:48:59 | gibi | there are plenty of hits not just in nova | |
| 10:50:01 | gibi | only affecting python 3.x jobs not the 2.7 jobs | |
| 10:55:04 | jkulik | https://github.com/networkx/networkx/commit/6b1ce03f485076d39994e8d624bbf6ca82466eb9#diff-027182481aebf9ad0dda6ca00714653aR95 seems to be the cause | |
| 10:56:00 | jkulik | seems like there's no upper version requirement defined in the taskflow library for python3 https://opendev.org/openstack/taskflow/src/branch/master/requirements.txt#L24 | |
| 10:59:41 | gibi | jkulik: good find. I agree that this can be the problem | |
| 11:06:11 | frickler | gibi: jkulik: that matches what I found comparing pip freeze from working and broken jobs. want to propose a cap to reqs as a quick fix? | |
| 12:42:32 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| 12:44:28 | brinzhang | mriedem: There is an issue, do you have time to review https://review.opendev.org/#/c/688635/5/novaclient/tests/functional/v2/test_migrations.py@116 | |
| 12:45:20 | brinzhang | mriedem: I am not find the way to get the in-progress live migration server to do the server-migration-list/show | |
| 12:45:48 | gibi | frickler jkulik: sorry I was pulled into a meeting in the meanwhile | |
| 12:46:57 | gibi | frickler: at the end I think taskflow needs to be fixed or the networkx req in taskflow needs to be pinned | |
| 12:47:06 | mriedem | brinzhang: likely not going to be able to do that one since it's not deterministic | |
| 12:47:42 | frickler | gibi: yes, updating taskflow not to use the deprecated attribute anymore would be the correct permanant solution I think | |
| 12:48:10 | frickler | gibi: also the u-c update here is failing, so this currently should only affect "special" jobs https://review.opendev.org/689079 | |
| 12:48:50 | brinzhang | mriedem: should I remove the server-migraton tests from this patch ? and todo it by follow-up in novaclient ? | |
| 12:50:17 | gibi | frickler: hm, but if upper-contraints pins networkx to 2.3 then how can be that py3.7 jobs are pulling in networkx 2.4? | |
| 12:50:43 | brinzhang | mriedem: If I make fake data, I don't think it's necessary to do this test. | |
| 12:51:31 | frickler | gibi: might be jobs ignoring u-c? didn't check in detail yet. | |
| 12:51:36 | mriedem | brinzhang: yeah i suppose, i forgot that those are only for in-progress live migrations | |
| 12:51:45 | mriedem | we can't do those anyway since the functional job is single-node | |
| 12:51:56 | brinzhang | yes | |
| 12:53:16 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| 12:54:28 | brinzhang_ | mriedem: I was updated this patch. | |
| 12:54:54 | mriedem | ok | |
| 12:56:44 | brinzhang_ | mriedem: thanks. | |
| 12:59:04 | gibi | frickler: locally when I reproduce the problem I see the following in the tox log | |
| 12:59:07 | gibi | $ cat .tox/py37/log/py37-1.log | grep networkx | |
| 12:59:09 | gibi | Ignoring networkx: markers 'python_version == "2.7"' don't match your environment | |
| 12:59:12 | gibi | Ignoring networkx: markers 'python_version == "3.4"' don't match your environment | |
| 12:59:15 | gibi | Ignoring networkx: markers 'python_version == "3.5"' don't match your environment | |
| 12:59:18 | gibi | Ignoring networkx: markers 'python_version == "3.6"' don't match your environment | |
| 13:00:19 | gibi | frickler: nvm, I use py3.7 so those logs are valid | |
| 13:00:21 | frickler | gibi: IIUC that's expected when running with python3.7, but is there a cap with ==3.7 in place? | |
| 13:00:54 | frickler | gibi: which repo are you running this in, nova or taskflow? | |
| 13:11:32 | mriedem | are the powervm driver unit tests failing since yesterday a known issue? | |
| 13:11:36 | mriedem | https://c6fecb2db5c55fa0effa-6229cc6450d9b491384804026d2fbd81.ssl.cf5.rackcdn.com/688980/1/gate/openstack-tox-py36/71a8bdd/testr_results.html.gz | |
| 13:11:50 | frickler | mriedem: IIUC that's the networkx issue | |
| 13:12:05 | frickler | gibi: seems transitive upper-constraints are ignored | |
| 13:12:18 | frickler | see e.g. https://zuul.opendev.org/t/openstack/build/2c6410c8f6344d19b9c88844b93f0683/log/job-output.txt#525-528 | |
| 13:12:33 | mriedem | yeah 2.4 was released 11 hours ago | |
| 13:15:08 | mriedem | https://bugs.launchpad.net/nova/+bug/1848499 | |
| 13:15:08 | openstack | Launchpad bug 1848499 in OpenStack Compute (nova) "powervm driver tests fail with networkx 2.4: "AttributeError: 'DiGraph' object has no attribute 'node'"" [Critical,Confirmed] | |
| 13:15:08 | efried | mriedem: do we need to cap taskflow? | |
| 13:15:17 | efried | oh, or networkx | |
| 13:15:27 | mriedem | it is capped in upper-constraints, | |
| 13:15:41 | mriedem | but as frickler said it seems it's not being honored | |
| 13:15:43 | frickler | if I add "networkx>=1.11" to nova/test-reqs.txt, the cap works. without it, it doesn't | |