Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-05
14:15:58 lyarwood mriedem: ah reproducer
14:16:19 mriedem yeah
14:16:21 lyarwood mriedem: no, that's fair it just looked like a trivial thing to change
14:16:36 mriedem ok i'd rather not backport it unless we have someone else reporting the problem
14:16:59 lyarwood mriedem: ack that's fine, I'll take a look next week at reproducing, if I can't I'll close
14:18:56 mriedem btw it occurs to me now we maybe shouldn't have merged this https://review.opendev.org/#/c/663887/
14:19:11 mriedem b/c i can't run fast8/pep8 on my bionic vm (py27 and py36) on rocky changes now
14:19:13 mriedem stephenfin: ^
14:21:40 stephenfin Because bionic doesn't support py3.5?
14:22:02 stephenfin It's a tough one. Without it, I can't run the same on my Fedora 30 machine for the reasons in the commit message
14:23:15 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add functional recreate test for regression bug 1825537 https://review.opendev.org/669361
14:23:15 openstack bug 1825537 in OpenStack Compute (nova) rocky "finish_resize failures incorrectly revert allocations" [Medium,Confirmed] https://launchpad.net/bugs/1825537
14:23:18 mriedem rocky doesn't support py37 at all though right?
14:23:28 mriedem rocky tox.ini at least has py36 targets
14:23:47 sean-k-mooney stephenfin: it use py36
14:23:51 mriedem iow, if you're doing rocky stuff and only have 3.7, you're going to need another vm
14:24:01 mriedem bionic has 2.7 and 3.6
14:24:12 stephenfin It doesn't, no, but the use of python3 means "give me any version of Python 3.x"
14:24:27 sean-k-mooney stephenfin: we dont support py3.5 anymore
14:24:34 sean-k-mooney its not part of the train PTI
14:24:40 stephenfin We did in rocky though
14:24:48 stephenfin That's a stable-only patch
14:24:48 sean-k-mooney rocky proably yes
14:25:40 stephenfin mriedem: Could we use basepython = python3.6 instead, assuming that's available on both Bionic and Xenial (or whatever 16.04 is?)
14:25:44 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Revert "tox: Use basepython of python3.5" https://review.opendev.org/669362
14:25:58 mriedem stephenfin: in rocky i don't think so,
14:26:12 mriedem because rocky supported (in the gate and the tox default env list) 27 and 35
14:26:17 mriedem but has a 36 target
14:26:39 mriedem there is no official 37 test targeting in rocky though
14:27:04 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Boot a VM with an unaddressed port https://review.opendev.org/641670
14:27:06 mriedem so like i said, i think if you're doing rocky changes and only have 37, you'll just need another dev vm for working on those older branches (which is what i have to do for ocata changes at this point)
14:27:12 mriedem i have a xenial vm just for doing ocata stuff now
14:27:44 mriedem xenial less for the python thing but more b/c of native packages like postgresql
15:04:53 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove nova-consoleauth https://review.opendev.org/652969
15:07:27 stephenfin lyarwood: The stable/stein variant of this has merged now. Could you tackle this one? https://review.opendev.org/#/c/667355/
15:20:58 dansmith sean-k-mooney: mriedem: Artom made several changes to the revert resize patch yesterday but didn't respond to comments and also rebased it so there's a lot of change.. did I miss any discussion about that?
15:21:28 dansmith looks like he removed the try..except around the signature change which was my deal, but otherwise I either need to eye-diff it or pull it down
15:21:46 sean-k-mooney dansmith: he change it to the db lookup
15:21:53 sean-k-mooney right
15:22:02 sean-k-mooney i havent looked at it this mornig
15:22:09 dansmith alright
15:23:58 mriedem i was going to have to eye-diff anyway
15:34:07 sean-k-mooney artom is on pto while he is flying home to canada today but if there are small things like mising test exctra ill be around for an hour or two to fix them
15:34:38 sean-k-mooney i think artom is back on monday but if not ill continue looking at this until he is back.
15:55:03 dansmith sean-k-mooney: I don't see the "will wait for bind time events" string in the logs
15:55:21 dansmith on the hybrid plug test in experimental
15:55:51 dansmith maybe that test is single-host which means we're not running any of this new stuff anyway?
15:56:12 sean-k-mooney it is
15:56:22 sean-k-mooney it is testign the resize to same host case
15:56:42 dansmith ah yeah I see the same host debug message instead
15:56:44 sean-k-mooney https://review.opendev.org/#/c/664442/14 is testing resize to different host
15:57:28 dansmith ah, right forgot that was on top and was thinking we'd get testing from mriedem's patch, I see
15:58:24 sean-k-mooney ya this is the message form the dnm change http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_49_57_371761
15:59:02 dansmith yeah, but lower we receive and discard that event: http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_49_58_003384
15:59:22 dansmith see the "no event matching" above that
15:59:22 sean-k-mooney ya i think this might be the souce node i need to check the other node
15:59:29 sean-k-mooney and confim this is the right vm
15:59:41 dansmith we shouldn't emit the "will wait..." message on the wrong node right?
16:00:13 sean-k-mooney we should not e might get thos events on both nodes
16:00:23 dansmith receive, but not waiting
16:00:41 dansmith actually, that's the unplug event
16:00:48 dansmith we get the plug event shortly after that:
16:01:02 dansmith http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_50_00_983164
16:02:55 sean-k-mooney i dont kno if we ever actully wait for unplugged events do we
16:08:00 sean-k-mooney it looks like libvirt died oh that vm breilfy http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_59_12_862599
16:08:16 sean-k-mooney which is unrelated but annoying
16:08:48 sean-k-mooney actully that is part of the testing i think
16:09:03 sean-k-mooney i think we are killing it so we can do evacuate testing
16:15:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: WIP: Drop source node allocations if finish_resize fails https://review.opendev.org/669393
16:17:03 mriedem the nova-live-migration job, renamed to nova-multinode in 664442, yes does evacuate testing and kills libvirtd on the controller node
16:19:03 sean-k-mooney in any case we start waiting for the event here http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_48_754192 and we recive it here http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_51_905792
16:20:00 sean-k-mooney there is a vif unplugged and plugged event recrved just before we wait that im not sure where they came form
16:20:25 sean-k-mooney http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_40_441961 and http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_45_258629
16:21:05 sean-k-mooney but it does look like we do wait as expected
16:31:08 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Drop source node allocations if finish_resize fails https://review.opendev.org/669393
16:31:24 mriedem lyarwood: dansmith: ^ this backport got a bit hairy because before stein we had to deal with non-migration-based allocations,
16:31:44 mriedem so i've documentd the difference in the commit message and code, and added a functional test for the non-migration-based allocations legacy wrinkle
16:31:55 mriedem just fyi so it doesn't get fast-approved as a stable core
16:35:24 dansmith ugh
17:18:37 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova master: [WIP] Allow to spawn VMs with port without IP address https://review.opendev.org/669411
17:22:09 redkrieg I'm having an interesting issue with a large flavor not being deployable on a compute host but two flavors of half the size working fine. the host has just enough disk and ram for the flavor, plus disk_allocation_ratio and ram_allocation_ratio are > 1. Nova scheduler logs with debug enabled don't even show the host in question during filtering, so I think it's being removed earlier somehow. I
17:22:15 redkrieg see accurate information in the "Update host state from compute node" ComputeNode output just minutes before launching an instance. Can anyone suggest a reason it wouldn't even be considered for filtering?
17:37:24 redkrieg I've narrowed it down to a ram issue by dropping my flavor's ram requirement to 30000 from 32768. the logs show that the hypervisor is reporting 32141 but even 32000 doesn't work. Is there some hard limit I don't know about that bypasses the allocation ratio settings?
17:58:40 openstackgerrit Corey Bryant proposed openstack/nova master: Add Python 3 Train unit tests https://review.opendev.org/669437
18:00:20 openstackgerrit Corey Bryant proposed openstack/os-vif master: Add Python 3 Train unit tests https://review.opendev.org/669438
18:01:06 openstackgerrit Corey Bryant proposed openstack/python-novaclient master: Add Python 3 Train unit tests https://review.opendev.org/669439
18:08:50 mriedem redkrieg: which release? placement could be filtering it out - enable debug logging in the placement logs and you should see the allocation candidate filtering messages (on newer releases), there is the reserved_host_ram_mb config option on the compute that will be reporting reserved MEMORY_MB to placement
18:08:57 mriedem which is part of the overall capacity calculation
18:12:20 Nick_A mriedem - rocky, phys_ram=32141MB used_ram=0MB, reserved_host_memory_mb = 0 in nova.conf
18:12:44 Nick_A we found https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html which explains why 32768 doesn't work, but we're not seeing why 32000 is also failing
18:17:27 mriedem are you using the RamFilter or AggregateRamFilter?
18:21:55 redkrieg yes, we do have the AggregateRamFilter active. I don't see this host being filtered by it though.
18:25:07 openstackgerrit Merged openstack/nova master: libvirt: Remove MIN_{QEMU,LIBVIRT}_LUKS_VERSION https://review.opendev.org/668924
18:31:18 openstackgerrit Merged openstack/nova-specs master: Fix the cross_az_attach value in AZ-unshelved spec https://review.opendev.org/669261
18:34:48 mriedem the reason i ask is because those filters return a "limit" from the scheduler which gets passed down to the compute which is part of the resource claim,
18:35:00 mriedem and if the scheduler is passing down a limit for something that you don't expect, that might be the cause
18:35:39 mriedem https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios
18:37:17 mriedem iow, what you think and what placement is using might be different allocation ratios
18:37:28 mriedem hopefully that doc gives some things to look at
18:48:34 Nick_A It actually looks like we're dealing with some sort of qemu/kernel bug... It successfully attempts a claim on compute if we set the flavor RAM less than the physical RAM, but libvirt errors at certain amounts with "qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory."

Earlier   Later