Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-25
21:12:50 sean-k-mooney efried: has intel not given you server with 70 billion cores to run cirros vms on yet
21:13:09 efried pshht, what do you think?
21:13:10 sean-k-mooney they are greate for builing docs or running unit tests
21:13:47 sean-k-mooney i had 3 racks woth of severs in my name when i left so yes?
21:13:59 efried even if I had 70 billion cores, I would still want to build only the spec I modified.
21:14:35 sean-k-mooney you could add that script to the tools directly and add it as a tox target
21:14:37 mriedem eric is in HMC withdrawals
21:14:41 sean-k-mooney like fast8
21:15:07 efried that's a good idea.
21:16:50 efried where do those tools live?
21:17:17 mriedem they are just scripts in the repo
21:17:22 sean-k-mooney here https://github.com/openstack/nova-specs/tree/master/tools
21:17:24 mriedem the tox target calls them and passes through the args
21:17:32 efried oh, I thought there was a central repo
21:17:37 sean-k-mooney no
21:17:46 sean-k-mooney we copy past all the things
21:18:04 sean-k-mooney there is a cookiecutter template somwhere
21:18:15 mriedem oslo-specs-incubator duh
21:18:21 sean-k-mooney but this would likely only be for tests altherough i guess wyou could use it in nova
21:19:05 sean-k-mooney i still fine the oslo incubator graduation sript to be magical
21:19:26 sean-k-mooney we should have used it for placement extraction actully but heindsight
21:20:03 mriedem i was joking
21:20:20 mriedem i just like to make oslo-incubator jokes to feel worldly
21:20:28 mriedem feel like a BIG MAN
21:20:58 sean-k-mooney sure but i really liked this script https://github.com/openstack/oslo-incubator/blob/stable/kilo/tools/filter_git_history.sh
21:22:00 sean-k-mooney i also really hated it after the 4 or 5th time i imported chagne form neutron into networking-ovs-dpdk
21:22:11 sean-k-mooney but it was nic to be able to maintain the history
21:24:03 openstackgerrit Eric Fried proposed openstack/nova master: Process [compute] in $NOVA_CPU_CONF in nova-next https://review.opendev.org/672800
21:26:40 efried mriedem: Should nova-cpu.conf need [[api_]database]/connection ?
21:27:14 sean-k-mooney no...
21:27:33 sean-k-mooney i dont think it should but mriedem or dansmith would know
21:28:31 mriedem efried: no
21:29:04 mriedem whether or not the cell conductor service needs to hit the api db depends on if you're allowing up-calls
21:29:09 mriedem which most people are
21:29:15 mriedem because we haven't closed all of those gaps
21:29:27 mriedem https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls
21:30:59 openstack Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)
21:30:59 sean-k-mooney by the way i was talking to slaweq this morning about https://bugs.launchpad.net/neutron/+bug/1836642 and im pretty sure we root cased this to the fact that nova is not confgiured to use memcache in the gate and we are falling back to using the dogpile.null cache implementaion
21:31:16 efried red herring
21:31:24 efried (db conns from compute)
21:32:27 mriedem sean-k-mooney: coincidentally i just saw he added me to https://review.opendev.org/#/c/672715/1
21:32:33 openstack Launchpad bug 1808010 in OpenStack Compute (nova) "Tempest cirros ssh setup fails due to lack of disk space causing config-drive setup to fail forcing fallback to metadata server which fails due to hitting 10 second timeout." [Medium,Confirmed]
21:32:33 sean-k-mooney we are hitting ^ downstream too so in at least the donwnstream case i dont think its a duplicate of https://bugs.launchpad.net/openstack-gate/+bug/1808010
21:33:45 sean-k-mooney mriedem: oh cool ya just wanted to give ye an fyi as he said he was working on a fix in tempest/devstack
22:15:51 openstackgerrit melanie witt proposed openstack/nova stable/stein: Avoid logging traceback when detach device not found https://review.opendev.org/672833
22:22:42 openstackgerrit sean mooney proposed openstack/os-vif master: only disable mac ageing for ovs hybrid plug https://review.opendev.org/672834
22:28:34 eandersson Anyone seen VMs failed scheduling getting stuck in building/scheduling indefinitely?
22:29:11 eandersson > There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for
22:29:13 eandersson > MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance
22:29:35 eandersson Hit a scheduling race condition which is fine, but then just got stuck (never into an error'd state)
22:29:44 sean-k-mooney it should not get stuc indefinetly and go to an error state
22:29:58 eandersson Yea - exactly
22:30:26 eandersson We have seen this a few times when something like Senlin is aggressively scaling up
22:31:25 eandersson And of course Senlin is unhappy because it just sits there waiting for it to go into ACTIVE (or ERROR'd) state.
22:31:29 sean-k-mooney if you are using any numa related resouces like hugepages or cpu pinning or if you are using sriov/pci passthough these are not tracked in placenet so if you have more then 1 schduler with more then 1 worker they can race
22:31:49 eandersson Yea - that is exactly it.
22:32:49 eandersson We are fine with it failing due to the race condition.
22:33:00 sean-k-mooney if we make it to 2020 then this should be fixed in U when all that stuff is in placmenet. but back to your current problem are there any error in the conductor that could indicate why the instance was not put into error state
22:33:20 eandersson Nope just the above errors.
22:34:27 eandersson Starts with the expected
22:34:29 eandersson > Free vcpu 0.00 VCPU < requested 20 VCPU
22:34:51 sean-k-mooney what release are you running?
22:35:26 sean-k-mooney this is where that error is being raised by the way https://github.com/openstack/nova/blob/3370f0f03ce17aaf3a7ebaa95d497f62bef238c0/nova/conductor/manager.py#L630
22:35:31 eandersson Rocky
22:35:58 sean-k-mooney have you disabled the core,ram and disk filters
22:36:04 eandersson We have not
22:36:25 sean-k-mooney http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html
22:36:38 sean-k-mooney they have been deprecated and should not be enabled after ocata
22:37:30 sean-k-mooney we stopped reporting the info it used i think in rocky or stien
22:38:02 sean-k-mooney so it might not have been a race we could have elimiated all the host because the filter did not work
22:40:40 eandersson I feel like that would have been a more visible problem thou
22:40:54 eandersson We are seeing this happen in %0.1>
22:41:15 sean-k-mooney ya i think if we got to this part of the code its not the filters
22:41:34 sean-k-mooney you should turn them off however as they will break when you upgade to stien
22:42:23 sean-k-mooney so after we log that message we shoudl raise here https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L601
22:42:50 sean-k-mooney and end up just below it here https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L615
22:43:22 sean-k-mooney at which port we shoudl notify that the vm state should be error
22:47:39 sean-k-mooney https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/utils.py#L573 is where teh exception is being loged as a waning
22:47:54 sean-k-mooney and we do an instance.save right after it
22:48:19 sean-k-mooney that save should have updated the instance to error
22:48:49 eandersson Does overcommit no longer work in aggregates?
22:49:10 sean-k-mooney we broke that
22:49:29 sean-k-mooney in ocata
22:51:30 sean-k-mooney from ocata on you need to set the overcommit per node
22:51:37 sean-k-mooney in stien we implemneted https://github.com/openstack/nova-specs/blob/master/specs/stein/implemented/initial-allocation-ratios.rst
22:51:56 sean-k-mooney which allow you to manage allocation ratios directly via placement
22:52:09 sean-k-mooney and only specifiy an initall option in teh config
22:53:21 sean-k-mooney eandersson: but yes that is why melwitt sent http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html and why we deprecated teh core,ram and disk filters and there aggrate* conterparts
22:56:11 eandersson We might have placement slightly misconfigured
22:57:26 mriedem https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios on the scheduler allocation ratio + placement config stuff
22:57:48 mriedem the initial* options are only new in stein
22:57:52 mriedem so that doesn't help you in rocky
22:58:31 sean-k-mooney eandersson: in rocky the nova compute agent will continuoly set the ratios back to whatever is in the compute node config
22:58:32 melwitt eandersson: >= ocata there's no notion of a per aggregate allocation ratio. so you have to set them separately per compute host nova.conf
22:58:32 mriedem you can either override allocation ratios per compute or override the providers in placement...but i think compute will overwrite anything you set out of band
22:58:38 sean-k-mooney or use the defualt in code if not set
22:59:08 eandersson We set the computes wrong for non-overcommited at the moment
22:59:17 eandersson That could be causing the issue
22:59:45 mriedem note that you might not even be getting this far https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L619
22:59:51 sean-k-mooney its tricky becaue it will inially appar to work fine until you start filling your hosts

Earlier   Later