Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-17
09:53:09 sean-k-mooney its just reciving it
09:53:14 eandersson yea
09:53:28 sean-k-mooney did you also check with the instance uuid
09:53:34 eandersson I bet this compute is bust, but everything seems fine (including already existing vms)
09:53:39 eandersson yea
09:53:59 eandersson I do get some logs with the instance id
09:54:06 eandersson but it's just the ones I posted earlier
09:54:20 sean-k-mooney ok
09:54:22 eandersson > Instance ... has been scheduled to this compute host, the scheduler has made an allocation against this compute node but the instance has yet to start. Skipping heal of allocation: ...
09:54:46 sean-k-mooney right that is the periodic task
09:55:04 sean-k-mooney that make sure the resouce useage on the host matches what we expect
09:55:44 sean-k-mooney that is normal to see between when the host has been selected in the db as the target for the instnace and the vm starting
09:57:34 eandersson Anyway bed time. It's 3AM.
09:57:51 eandersson Thanks for helping sean-k-mooney and gibi
09:58:08 sean-k-mooney eandersson: o/ get some rest
10:32:24 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635
10:33:31 openstackgerrit Merged openstack/nova stable/queens: Fix 'has_calls' method calls in unit tests https://review.opendev.org/677378
10:42:09 gibi it is only me or the powervm unit tests started failing on master?
10:44:29 gibi AttributeError: 'DiGraph' object has no attribute 'node'
10:47:55 gibi https://702b7e8f253d29e679a6-2fe3f6c342189909aad5220492fb4721.ssl.cf1.rackcdn.com/688387/5/check/openstack-tox-cover/2c6410c/testr_results.html.gz
10:48:51 gibi http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttributeError%3A%20'DiGraph'%20object%20has%20no%20attribute%20'node'%5C%22&from=7d
10:48:59 gibi there are plenty of hits not just in nova
10:50:01 gibi only affecting python 3.x jobs not the 2.7 jobs
10:55:04 jkulik https://github.com/networkx/networkx/commit/6b1ce03f485076d39994e8d624bbf6ca82466eb9#diff-027182481aebf9ad0dda6ca00714653aR95 seems to be the cause
10:56:00 jkulik seems like there's no upper version requirement defined in the taskflow library for python3 https://opendev.org/openstack/taskflow/src/branch/master/requirements.txt#L24
10:59:41 gibi jkulik: good find. I agree that this can be the problem
11:06:11 frickler gibi: jkulik: that matches what I found comparing pip freeze from working and broken jobs. want to propose a cap to reqs as a quick fix?
12:42:32 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635
12:44:28 brinzhang mriedem: There is an issue, do you have time to review https://review.opendev.org/#/c/688635/5/novaclient/tests/functional/v2/test_migrations.py@116
12:45:20 brinzhang mriedem: I am not find the way to get the in-progress live migration server to do the server-migration-list/show
12:45:48 gibi frickler jkulik: sorry I was pulled into a meeting in the meanwhile
12:46:57 gibi frickler: at the end I think taskflow needs to be fixed or the networkx req in taskflow needs to be pinned
12:47:06 mriedem brinzhang: likely not going to be able to do that one since it's not deterministic
12:47:42 frickler gibi: yes, updating taskflow not to use the deprecated attribute anymore would be the correct permanant solution I think
12:48:10 frickler gibi: also the u-c update here is failing, so this currently should only affect "special" jobs https://review.opendev.org/689079
12:48:50 brinzhang mriedem: should I remove the server-migraton tests from this patch ? and todo it by follow-up in novaclient ?
12:50:17 gibi frickler: hm, but if upper-contraints pins networkx to 2.3 then how can be that py3.7 jobs are pulling in networkx 2.4?
12:50:43 brinzhang mriedem: If I make fake data, I don't think it's necessary to do this test.
12:51:31 frickler gibi: might be jobs ignoring u-c? didn't check in detail yet.
12:51:36 mriedem brinzhang: yeah i suppose, i forgot that those are only for in-progress live migrations
12:51:45 mriedem we can't do those anyway since the functional job is single-node
12:51:56 brinzhang yes
12:53:16 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Add functional test for migration-list in v2.80 https://review.opendev.org/688635
12:54:28 brinzhang_ mriedem: I was updated this patch.
12:54:54 mriedem ok
12:56:44 brinzhang_ mriedem: thanks.
12:59:04 gibi frickler: locally when I reproduce the problem I see the following in the tox log
12:59:07 gibi $ cat .tox/py37/log/py37-1.log | grep networkx
12:59:09 gibi Ignoring networkx: markers 'python_version == "2.7"' don't match your environment
12:59:12 gibi Ignoring networkx: markers 'python_version == "3.4"' don't match your environment
12:59:15 gibi Ignoring networkx: markers 'python_version == "3.5"' don't match your environment
12:59:18 gibi Ignoring networkx: markers 'python_version == "3.6"' don't match your environment
13:00:19 gibi frickler: nvm, I use py3.7 so those logs are valid
13:00:21 frickler gibi: IIUC that's expected when running with python3.7, but is there a cap with ==3.7 in place?
13:00:54 frickler gibi: which repo are you running this in, nova or taskflow?
13:11:32 mriedem are the powervm driver unit tests failing since yesterday a known issue?
13:11:36 mriedem https://c6fecb2db5c55fa0effa-6229cc6450d9b491384804026d2fbd81.ssl.cf5.rackcdn.com/688980/1/gate/openstack-tox-py36/71a8bdd/testr_results.html.gz
13:11:50 frickler mriedem: IIUC that's the networkx issue
13:12:05 frickler gibi: seems transitive upper-constraints are ignored
13:12:18 frickler see e.g. https://zuul.opendev.org/t/openstack/build/2c6410c8f6344d19b9c88844b93f0683/log/job-output.txt#525-528
13:12:33 mriedem yeah 2.4 was released 11 hours ago
13:15:08 efried mriedem: do we need to cap taskflow?
13:15:08 openstack Launchpad bug 1848499 in OpenStack Compute (nova) "powervm driver tests fail with networkx 2.4: "AttributeError: 'DiGraph' object has no attribute 'node'"" [Critical,Confirmed]
13:15:08 mriedem https://bugs.launchpad.net/nova/+bug/1848499
13:15:17 efried oh, or networkx
13:15:27 mriedem it is capped in upper-constraints,
13:15:41 mriedem but as frickler said it seems it's not being honored
13:15:43 frickler if I add "networkx>=1.11" to nova/test-reqs.txt, the cap works. without it, it doesn't
13:17:50 openstackgerrit Huachang Wang proposed openstack/nova master: To create single NUMA node instance in function '_get_numa_topology_auto' https://review.opendev.org/688932
13:17:51 openstackgerrit Huachang Wang proposed openstack/nova master: Add a new instance CPU allocation policy: mixed https://review.opendev.org/688934
13:17:51 openstackgerrit Huachang Wang proposed openstack/nova master: Assign and track instance pinning cpu through 'cpu_pinning' field https://review.opendev.org/688933
13:17:52 openstackgerrit Huachang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
13:17:52 openstackgerrit Huachang Wang proposed openstack/nova master: virt/libvirt: Get host pin cpuset according instance cpu_pinning https://review.opendev.org/688935
13:24:21 gibi frickler: it is the nova repo I'm using to reproduce
13:27:36 gibi frickler, mriedem: If I add '+ -r{toxinidir}/requirements.txt'
13:27:53 gibi frickler, mriedem: to tox.ini then the transitive deps are correct and the test passes
13:28:21 mriedem stephenfin: ^?
13:28:37 mriedem looks like networkx 2.4 changed Graph.node to Graph.nodes, weeee https://github.com/networkx/networkx/blob/networkx-2.4/doc/release/release_2.4.rst#deprecations
13:30:22 stephenfin looking
13:30:46 mriedem gibi: i want to say we used to explicitly include -r{toxinidir}/requirements.txt in deps but it was removed b/c it's installed for us in the tox env
13:30:54 mriedem but maybe that breaks u-c processing?
13:31:19 openstackgerrit Balazs Gibizer proposed openstack/nova master: Make sure tox install requirements.txt with upper-constraints https://review.opendev.org/689152
13:31:40 gibi mriedem, frickler, stephenfin: ^^ temporary fix
13:31:41 stephenfin mriedem, gibi: We need mordred/smcginnis for this
13:31:49 frickler gibi: mriedem: stephenfin: that seems a bug introduced in https://review.opendev.org/#/c/684775/
13:31:59 stephenfin gibi: You're essentially reverting b13c33caa07fc82b19c233f9ad46a1813eb3e76d
13:32:13 stephenfin I think we should probably do that or 19a0bdfec454bd921b718e7dc49fe2673fa79b10
13:32:44 stephenfin smcginnis: we don't include 'requirements.txt' in deps because tox builds an sdist for us which will include said deps automagically
13:32:47 openstackgerrit Takashi NATSUME proposed openstack/nova stable/stein: Fix unit of hw_rng:rate_period https://review.opendev.org/689153
13:33:18 stephenfin however, I switched from overriding 'install_command' to providing constraints via '-c FILE' in deps
13:33:21 frickler stephenfin: but it seems to include uncapped reqs
13:33:21 mriedem stephenfin: you mean revert https://review.opendev.org/#/c/684775/ right?
13:33:35 gibi stephenfin: install_commands are used for every install step in tox while deps only used to install test-requirements ?
13:33:45 stephenfin gibi: Yeah, I think so
13:33:46 smcginnis stephenfin: This seems OK - https://review.opendev.org/#/c/689152/1/tox.ini
13:33:50 smcginnis That is what is done elsewhere.
13:33:57 openstackgerrit Takashi NATSUME proposed openstack/nova stable/rocky: Fix unit of hw_rng:rate_period https://review.opendev.org/689154
13:34:20 smcginnis Then the other jobs that need different deps, like the lower-constraints job, override "deps" to set the requirements it needs.
13:34:46 stephenfin smcginnis: Cool. I didn't know this would happen so I need to go make sure we haven't broken other projects
13:35:03 smcginnis As long as the -c isn't hard coded in the install_command, settings "deps" should be flexible enough to use different requirements for different jobs.

Earlier   Later