| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-17 | |||
| 10:55:04 | jkulik | https://github.com/networkx/networkx/commit/6b1ce03f485076d39994e8d624bbf6ca82466eb9#diff-027182481aebf9ad0dda6ca00714653aR95 seems to be the cause | |
| 10:56:00 | jkulik | seems like there's no upper version requirement defined in the taskflow library for python3 https://opendev.org/openstack/taskflow/src/branch/master/requirements.txt#L24 | |
| 10:59:41 | gibi | jkulik: good find. I agree that this can be the problem | |
| 11:06:11 | frickler | gibi: jkulik: that matches what I found comparing pip freeze from working and broken jobs. want to propose a cap to reqs as a quick fix? | |
| 12:42:32 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| 12:44:28 | brinzhang | mriedem: There is an issue, do you have time to review https://review.opendev.org/#/c/688635/5/novaclient/tests/functional/v2/test_migrations.py@116 | |
| 12:45:20 | brinzhang | mriedem: I am not find the way to get the in-progress live migration server to do the server-migration-list/show | |
| 12:45:48 | gibi | frickler jkulik: sorry I was pulled into a meeting in the meanwhile | |
| 12:46:57 | gibi | frickler: at the end I think taskflow needs to be fixed or the networkx req in taskflow needs to be pinned | |
| 12:47:06 | mriedem | brinzhang: likely not going to be able to do that one since it's not deterministic | |
| 12:47:42 | frickler | gibi: yes, updating taskflow not to use the deprecated attribute anymore would be the correct permanant solution I think | |
| 12:48:10 | frickler | gibi: also the u-c update here is failing, so this currently should only affect "special" jobs https://review.opendev.org/689079 | |
| 12:48:50 | brinzhang | mriedem: should I remove the server-migraton tests from this patch ? and todo it by follow-up in novaclient ? | |
| 12:50:17 | gibi | frickler: hm, but if upper-contraints pins networkx to 2.3 then how can be that py3.7 jobs are pulling in networkx 2.4? | |
| 12:50:43 | brinzhang | mriedem: If I make fake data, I don't think it's necessary to do this test. | |
| 12:51:31 | frickler | gibi: might be jobs ignoring u-c? didn't check in detail yet. | |
| 12:51:36 | mriedem | brinzhang: yeah i suppose, i forgot that those are only for in-progress live migrations | |
| 12:51:45 | mriedem | we can't do those anyway since the functional job is single-node | |
| 12:51:56 | brinzhang | yes | |
| 12:53:16 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635 | |
| 12:54:28 | brinzhang_ | mriedem: I was updated this patch. | |
| 12:54:54 | mriedem | ok | |
| 12:56:44 | brinzhang_ | mriedem: thanks. | |
| 12:59:04 | gibi | frickler: locally when I reproduce the problem I see the following in the tox log | |
| 12:59:07 | gibi | $ cat .tox/py37/log/py37-1.log | grep networkx | |
| 12:59:09 | gibi | Ignoring networkx: markers 'python_version == "2.7"' don't match your environment | |
| 12:59:12 | gibi | Ignoring networkx: markers 'python_version == "3.4"' don't match your environment | |
| 12:59:15 | gibi | Ignoring networkx: markers 'python_version == "3.5"' don't match your environment | |
| 12:59:18 | gibi | Ignoring networkx: markers 'python_version == "3.6"' don't match your environment | |
| 13:00:19 | gibi | frickler: nvm, I use py3.7 so those logs are valid | |
| 13:00:21 | frickler | gibi: IIUC that's expected when running with python3.7, but is there a cap with ==3.7 in place? | |
| 13:00:54 | frickler | gibi: which repo are you running this in, nova or taskflow? | |
| 13:11:32 | mriedem | are the powervm driver unit tests failing since yesterday a known issue? | |
| 13:11:36 | mriedem | https://c6fecb2db5c55fa0effa-6229cc6450d9b491384804026d2fbd81.ssl.cf5.rackcdn.com/688980/1/gate/openstack-tox-py36/71a8bdd/testr_results.html.gz | |
| 13:11:50 | frickler | mriedem: IIUC that's the networkx issue | |
| 13:12:05 | frickler | gibi: seems transitive upper-constraints are ignored | |
| 13:12:18 | frickler | see e.g. https://zuul.opendev.org/t/openstack/build/2c6410c8f6344d19b9c88844b93f0683/log/job-output.txt#525-528 | |
| 13:12:33 | mriedem | yeah 2.4 was released 11 hours ago | |
| 13:15:08 | mriedem | https://bugs.launchpad.net/nova/+bug/1848499 | |
| 13:15:08 | openstack | Launchpad bug 1848499 in OpenStack Compute (nova) "powervm driver tests fail with networkx 2.4: "AttributeError: 'DiGraph' object has no attribute 'node'"" [Critical,Confirmed] | |
| 13:15:08 | efried | mriedem: do we need to cap taskflow? | |
| 13:15:17 | efried | oh, or networkx | |
| 13:15:27 | mriedem | it is capped in upper-constraints, | |
| 13:15:41 | mriedem | but as frickler said it seems it's not being honored | |
| 13:15:43 | frickler | if I add "networkx>=1.11" to nova/test-reqs.txt, the cap works. without it, it doesn't | |
| 13:17:50 | openstackgerrit | Huachang Wang proposed openstack/nova master: To create single NUMA node instance in function '_get_numa_topology_auto' https://review.opendev.org/688932 | |
| 13:17:51 | openstackgerrit | Huachang Wang proposed openstack/nova master: Assign and track instance pinning cpu through 'cpu_pinning' field https://review.opendev.org/688933 | |
| 13:17:51 | openstackgerrit | Huachang Wang proposed openstack/nova master: Add a new instance CPU allocation policy: mixed https://review.opendev.org/688934 | |
| 13:17:52 | openstackgerrit | Huachang Wang proposed openstack/nova master: virt/libvirt: Get host pin cpuset according instance cpu_pinning https://review.opendev.org/688935 | |
| 13:17:52 | openstackgerrit | Huachang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | |
| 13:24:21 | gibi | frickler: it is the nova repo I'm using to reproduce | |
| 13:27:36 | gibi | frickler, mriedem: If I add '+ -r{toxinidir}/requirements.txt' | |
| 13:27:53 | gibi | frickler, mriedem: to tox.ini then the transitive deps are correct and the test passes | |
| 13:28:21 | mriedem | stephenfin: ^? | |
| 13:28:37 | mriedem | looks like networkx 2.4 changed Graph.node to Graph.nodes, weeee https://github.com/networkx/networkx/blob/networkx-2.4/doc/release/release_2.4.rst#deprecations | |
| 13:30:22 | stephenfin | looking | |
| 13:30:46 | mriedem | gibi: i want to say we used to explicitly include -r{toxinidir}/requirements.txt in deps but it was removed b/c it's installed for us in the tox env | |
| 13:30:54 | mriedem | but maybe that breaks u-c processing? | |
| 13:31:19 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make sure tox install requirements.txt with upper-constraints https://review.opendev.org/689152 | |
| 13:31:40 | gibi | mriedem, frickler, stephenfin: ^^ temporary fix | |
| 13:31:41 | stephenfin | mriedem, gibi: We need mordred/smcginnis for this | |
| 13:31:49 | frickler | gibi: mriedem: stephenfin: that seems a bug introduced in https://review.opendev.org/#/c/684775/ | |
| 13:31:59 | stephenfin | gibi: You're essentially reverting b13c33caa07fc82b19c233f9ad46a1813eb3e76d | |
| 13:32:13 | stephenfin | I think we should probably do that or 19a0bdfec454bd921b718e7dc49fe2673fa79b10 | |
| 13:32:44 | stephenfin | smcginnis: we don't include 'requirements.txt' in deps because tox builds an sdist for us which will include said deps automagically | |
| 13:32:47 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/stein: Fix unit of hw_rng:rate_period https://review.opendev.org/689153 | |
| 13:33:18 | stephenfin | however, I switched from overriding 'install_command' to providing constraints via '-c FILE' in deps | |
| 13:33:21 | mriedem | stephenfin: you mean revert https://review.opendev.org/#/c/684775/ right? | |
| 13:33:21 | frickler | stephenfin: but it seems to include uncapped reqs | |
| 13:33:35 | gibi | stephenfin: install_commands are used for every install step in tox while deps only used to install test-requirements ? | |
| 13:33:45 | stephenfin | gibi: Yeah, I think so | |
| 13:33:46 | smcginnis | stephenfin: This seems OK - https://review.opendev.org/#/c/689152/1/tox.ini | |
| 13:33:50 | smcginnis | That is what is done elsewhere. | |
| 13:33:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Fix unit of hw_rng:rate_period https://review.opendev.org/689154 | |
| 13:34:20 | smcginnis | Then the other jobs that need different deps, like the lower-constraints job, override "deps" to set the requirements it needs. | |
| 13:34:46 | stephenfin | smcginnis: Cool. I didn't know this would happen so I need to go make sure we haven't broken other projects | |
| 13:35:03 | smcginnis | As long as the -c isn't hard coded in the install_command, settings "deps" should be flexible enough to use different requirements for different jobs. | |
| 13:35:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/queens: Fix unit of hw_rng:rate_period https://review.opendev.org/689155 | |
| 13:36:10 | smcginnis | One tricky bit I've seen is that it's not always obvious that putting a deps line in a tox environment overrides rather than appends to the deps that are used. So there have been some cases where teams have meant to use an additional file but have ended up excluding some common ones. | |
| 13:36:31 | smcginnis | So just make sure wherever used deps you always include things like test-requirements (where appropriate of course). | |
| 13:36:40 | stephenfin | gibi: comments left | |
| 13:37:52 | stephenfin | smcginnis: Yeah, we're good there. We use the '{testenv[blah]}deps' syntax everywhere that matters | |
| 13:37:59 | mriedem | stephenfin: replied to you in there | |
| 13:38:47 | smcginnis | stephenfin: ++ | |
| 13:38:48 | stephenfin | mriedem: yup, both valid | |
| 13:39:42 | stephenfin | gibi, efried, mriedem, alex_xu: Also, I'm on PTO from tomorrow until the summit. Just FYI | |
| 13:40:20 | efried | holy crap, that's in like two and a half weeks, I didn't realize how close it was. | |
| 13:40:39 | sean-k-mooney | yep | |
| 13:40:43 | stephenfin | you're telling me | |
| 13:41:18 | efried | gdi, I have to take my kid to the dentist this morning during meeting time | |
| 13:41:31 | gibi | stephenfin: I've just confirmed that haveing the constraint in the install_command also works | |
| 13:41:59 | stephenfin | gibi: yeah, I think we don't want to do that because people forget to override install_command for the lower-constraints target | |
| 13:42:31 | stephenfin | at least that's what I took away from smcginnis' comments above and elsewhere | |
| 13:42:52 | smcginnis | Correct. | |
| 13:43:21 | gibi | stephenfin, smcginnis: I got your comments on the fix, I will respin that patch quickly | |
| 13:43:58 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/pike: Fix unit of hw_rng:rate_period https://review.opendev.org/689158 | |
| 13:46:33 | efried | okay, I'm going to be late to the meeting, but hopefully not more than a few minutes | |
| 13:46:49 | efried | poor planning (from like months ago) | |
| 13:50:31 | mriedem | i'll start the meeting | |
| 14:00:17 | mriedem | nova meeting starting now in -meeting | |