| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 20:42:28 | efried | uck | |
| 20:43:26 | efried | mriedem: but yeah, I think it makes sense to add this in there. | |
| 20:43:46 | mriedem | yeah | |
| 20:44:26 | mriedem | i frequently filter on migration_type and instance_uuid in functional tests since GET /servers/{server_id}/migrations is hard-coded to only be in-progress live migrations | |
| 20:54:56 | efried | mriedem: chuck that filter field in there and I'm +2 | |
| 20:55:09 | mriedem | alright then | |
| 20:55:13 | efried | not sure if you're still comfortable being the other +2 | |
| 20:55:21 | mriedem | o | |
| 20:55:28 | mriedem | *i'm like a 1.5 | |
| 20:56:08 | efried | sean-k-mooney: want to throw a +1 on there to push us over the edge? | |
| 20:56:13 | efried | ...once mriedem has updated | |
| 20:56:23 | sean-k-mooney | efried: link? | |
| 20:56:31 | efried | https://review.opendev.org/#/c/667894/ -- stand by for PS4 | |
| 20:57:06 | sean-k-mooney | oh i have looked at that before sure ill re review | |
| 21:05:12 | mriedem | build specs docs takes awhile | |
| 21:05:26 | mriedem | *building | |
| 21:05:47 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 21:05:47 | sean-k-mooney | im reviewing v3 in the mean time and then ill revew the delta | |
| 21:05:47 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP-until-series-is-ready] Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 21:05:48 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | |
| 21:05:48 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for sending NUMAMigrateData to the source https://review.opendev.org/634828 | |
| 21:05:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 21:05:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: RPC changes to prepare for NUMA live migration https://review.opendev.org/634605 | |
| 21:05:50 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Functional test for NUMA live migration https://review.opendev.org/672595 | |
| 21:05:50 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 21:06:16 | efried | mriedem: I use this: http://paste.openstack.org/raw/754874/ | |
| 21:08:39 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add user_id to the migrations https://review.opendev.org/667894 | |
| 21:10:40 | efried | oh, I retract my 'uck' from earlier. Didn't pick up that these were in the querystring. That's dandy. | |
| 21:11:55 | sean-k-mooney | ya that is how we normally do filtering | |
| 21:12:22 | mriedem | yeah i wasn't sure why you were ucking | |
| 21:12:50 | sean-k-mooney | efried: has intel not given you server with 70 billion cores to run cirros vms on yet | |
| 21:13:09 | efried | pshht, what do you think? | |
| 21:13:10 | sean-k-mooney | they are greate for builing docs or running unit tests | |
| 21:13:47 | sean-k-mooney | i had 3 racks woth of severs in my name when i left so yes? | |
| 21:13:59 | efried | even if I had 70 billion cores, I would still want to build only the spec I modified. | |
| 21:14:35 | sean-k-mooney | you could add that script to the tools directly and add it as a tox target | |
| 21:14:37 | mriedem | eric is in HMC withdrawals | |
| 21:14:41 | sean-k-mooney | like fast8 | |
| 21:15:07 | efried | that's a good idea. | |
| 21:16:50 | efried | where do those tools live? | |
| 21:17:17 | mriedem | they are just scripts in the repo | |
| 21:17:22 | sean-k-mooney | here https://github.com/openstack/nova-specs/tree/master/tools | |
| 21:17:24 | mriedem | the tox target calls them and passes through the args | |
| 21:17:32 | efried | oh, I thought there was a central repo | |
| 21:17:37 | sean-k-mooney | no | |
| 21:17:46 | sean-k-mooney | we copy past all the things | |
| 21:18:04 | sean-k-mooney | there is a cookiecutter template somwhere | |
| 21:18:15 | mriedem | oslo-specs-incubator duh | |
| 21:18:21 | sean-k-mooney | but this would likely only be for tests altherough i guess wyou could use it in nova | |
| 21:19:05 | sean-k-mooney | i still fine the oslo incubator graduation sript to be magical | |
| 21:19:26 | sean-k-mooney | we should have used it for placement extraction actully but heindsight | |
| 21:20:03 | mriedem | i was joking | |
| 21:20:20 | mriedem | i just like to make oslo-incubator jokes to feel worldly | |
| 21:20:28 | mriedem | feel like a BIG MAN | |
| 21:20:58 | sean-k-mooney | sure but i really liked this script https://github.com/openstack/oslo-incubator/blob/stable/kilo/tools/filter_git_history.sh | |
| 21:22:00 | sean-k-mooney | i also really hated it after the 4 or 5th time i imported chagne form neutron into networking-ovs-dpdk | |
| 21:22:11 | sean-k-mooney | but it was nic to be able to maintain the history | |
| 21:24:03 | openstackgerrit | Eric Fried proposed openstack/nova master: Process [compute] in $NOVA_CPU_CONF in nova-next https://review.opendev.org/672800 | |
| 21:26:40 | efried | mriedem: Should nova-cpu.conf need [[api_]database]/connection ? | |
| 21:27:14 | sean-k-mooney | no... | |
| 21:27:33 | sean-k-mooney | i dont think it should but mriedem or dansmith would know | |
| 21:28:31 | mriedem | efried: no | |
| 21:29:04 | mriedem | whether or not the cell conductor service needs to hit the api db depends on if you're allowing up-calls | |
| 21:29:09 | mriedem | which most people are | |
| 21:29:15 | mriedem | because we haven't closed all of those gaps | |
| 21:29:27 | mriedem | https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls | |
| 21:30:59 | openstack | Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq) | |
| 21:30:59 | sean-k-mooney | by the way i was talking to slaweq this morning about https://bugs.launchpad.net/neutron/+bug/1836642 and im pretty sure we root cased this to the fact that nova is not confgiured to use memcache in the gate and we are falling back to using the dogpile.null cache implementaion | |
| 21:31:16 | efried | red herring | |
| 21:31:24 | efried | (db conns from compute) | |
| 21:32:27 | mriedem | sean-k-mooney: coincidentally i just saw he added me to https://review.opendev.org/#/c/672715/1 | |
| 21:32:33 | openstack | Launchpad bug 1808010 in OpenStack Compute (nova) "Tempest cirros ssh setup fails due to lack of disk space causing config-drive setup to fail forcing fallback to metadata server which fails due to hitting 10 second timeout." [Medium,Confirmed] | |
| 21:32:33 | sean-k-mooney | we are hitting ^ downstream too so in at least the donwnstream case i dont think its a duplicate of https://bugs.launchpad.net/openstack-gate/+bug/1808010 | |
| 21:33:45 | sean-k-mooney | mriedem: oh cool ya just wanted to give ye an fyi as he said he was working on a fix in tempest/devstack | |
| 22:15:51 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Avoid logging traceback when detach device not found https://review.opendev.org/672833 | |
| 22:22:42 | openstackgerrit | sean mooney proposed openstack/os-vif master: only disable mac ageing for ovs hybrid plug https://review.opendev.org/672834 | |
| 22:28:34 | eandersson | Anyone seen VMs failed scheduling getting stuck in building/scheduling indefinitely? | |
| 22:29:11 | eandersson | > There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for | |
| 22:29:13 | eandersson | > MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance | |
| 22:29:35 | eandersson | Hit a scheduling race condition which is fine, but then just got stuck (never into an error'd state) | |
| 22:29:44 | sean-k-mooney | it should not get stuc indefinetly and go to an error state | |
| 22:29:58 | eandersson | Yea - exactly | |
| 22:30:26 | eandersson | We have seen this a few times when something like Senlin is aggressively scaling up | |
| 22:31:25 | eandersson | And of course Senlin is unhappy because it just sits there waiting for it to go into ACTIVE (or ERROR'd) state. | |
| 22:31:29 | sean-k-mooney | if you are using any numa related resouces like hugepages or cpu pinning or if you are using sriov/pci passthough these are not tracked in placenet so if you have more then 1 schduler with more then 1 worker they can race | |
| 22:31:49 | eandersson | Yea - that is exactly it. | |
| 22:32:49 | eandersson | We are fine with it failing due to the race condition. | |
| 22:33:00 | sean-k-mooney | if we make it to 2020 then this should be fixed in U when all that stuff is in placmenet. but back to your current problem are there any error in the conductor that could indicate why the instance was not put into error state | |
| 22:33:20 | eandersson | Nope just the above errors. | |
| 22:34:27 | eandersson | Starts with the expected | |
| 22:34:29 | eandersson | > Free vcpu 0.00 VCPU < requested 20 VCPU | |
| 22:34:51 | sean-k-mooney | what release are you running? | |
| 22:35:26 | sean-k-mooney | this is where that error is being raised by the way https://github.com/openstack/nova/blob/3370f0f03ce17aaf3a7ebaa95d497f62bef238c0/nova/conductor/manager.py#L630 | |
| 22:35:31 | eandersson | Rocky | |
| 22:35:58 | sean-k-mooney | have you disabled the core,ram and disk filters | |
| 22:36:04 | eandersson | We have not | |
| 22:36:25 | sean-k-mooney | http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html | |
| 22:36:38 | sean-k-mooney | they have been deprecated and should not be enabled after ocata | |
| 22:37:30 | sean-k-mooney | we stopped reporting the info it used i think in rocky or stien | |
| 22:38:02 | sean-k-mooney | so it might not have been a race we could have elimiated all the host because the filter did not work | |