Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-21
13:51:10 efried I can't find anything in the code that's joining stderr to stdout. Unless the job itself is doing that.
13:52:17 efried Okay, apparently I'm the only one concerned about the fact that `openstack resource provider list` is producing bogus output in gmann's case, so I must be misunderstanding what we're actually trying to solve here. /me stfu, call if you need me.
13:53:48 gmann sean-k-mooney: and there it is working fine? failure case of octavia grenade job on py3.
13:54:36 sean-k-mooney they do that in code not via osc
13:54:47 gmann ohk
13:59:09 sean-k-mooney is looking up the provider by hostname all that is breakign the job?
13:59:57 sean-k-mooney it would be quick to fix osc-plamcenet to support that but equally quick to just do it with curl
14:01:32 gmann sean-k-mooney: fixing osc might take time with release etc until octavia job can install it from source.
14:01:54 sean-k-mooney do we know what cause the broken pipes?
14:02:03 gmann no.
14:02:21 haleyb sean-k-mooney: would adding osc-placement to requirements in octavia fix it as well?
14:02:53 sean-k-mooney i think mriedem said list does not currently support --name
14:04:02 sean-k-mooney i was looking at the greand job logs by the way gmann haleyb do you have the link to the octavia job
14:04:33 johnsom haleyb Yes, that is the error output you are seeing. Installing osc-placement should fix it.
14:04:49 gmann sean-k-mooney: https://review.opendev.org/#/c/693486/
14:04:50 johnsom https://www.irccloud.com/pastebin/PrDd7zG6/
14:05:08 openstackgerrit Merged openstack/nova stable/pike: Delete instance_id_mappings record in instance_destroy https://review.opendev.org/684658
14:05:21 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Ussuri" https://review.opendev.org/695056
14:05:42 sean-k-mooney ah right yes "'resource provider list -f value' is not an openstack command"
14:05:44 kashyap stephenfin: ^^ Fixed the functional test (forgot to run `tox -e functional-36`, bad me)
14:05:53 johnsom haleyb The question is which devstack plugin has the missing requirement
14:05:55 sean-k-mooney is becasue osc-placement is not installed
14:05:59 gmann johnsom: haleyb it is parsing of command failing not about osc command is need more installation etc
14:06:03 kashyap stephenfin: Thanks for the earlier review :-)
14:06:53 kashyap Uh, seems like I need a rebase...
14:07:04 sean-k-mooney johnsom: i guess the ocatavia one
14:07:17 gmann thus command we need to adjust to get the first RP- https://github.com/openstack/grenade/blob/fad62595bff3ae55b5b428a5ea00a9a168390fd2/projects/60_nova/resources.sh#L57
14:07:30 sean-k-mooney but honestly it might be better for devstack to install osc-placemetn if placement is installed
14:07:32 johnsom sean-k-mooney Octavia devstack plugin isn't running that command.
14:07:40 gmann command i mean parsing logic
14:07:51 sean-k-mooney oh ok
14:08:32 johnsom It is openstack/grenade
14:08:44 johnsom projects/60_nova/resources.sh
14:08:49 johnsom line 57
14:08:51 sean-k-mooney gmann: im confused on the octaiva patch you are linink to grenade
14:09:05 sean-k-mooney is this a but there are other fialng jobs too
14:09:14 sean-k-mooney *failing jobs
14:09:27 sean-k-mooney are ye only looking at the grenade one at the moment
14:10:02 gmann yeah grenade one only i checked
14:10:39 sean-k-mooney well we could have grenade install it or as i said we could have devstack install it if placemnt is installed
14:11:12 sean-k-mooney grenade is using it so its resonable for it to install its own depencies
14:11:20 johnsom Yeah. It's odd that grenade doesn't have a requirements.txt though it obvious has requirements in it's scripts
14:11:46 sean-k-mooney well grenade is not a python porject
14:12:39 sean-k-mooney its almost all bash so you would not pip install it
14:13:23 gmann osc-placement installation is not the issue here.
14:13:26 johnsom Ha, yeah, I just noticed that. I guess you know now how much time I have looked at grenade.... lol
14:14:09 johnsom gmann Yes it is. If that is not installed the first output of OSC is "openstack:" which the script tries to parse.
14:15:31 sean-k-mooney right as your irccloud link show the message is "openstack: 'resource provider list -f value' is not an openstack command. See 'openstack --help'."
14:15:47 johnsom Yep
14:17:13 gmann https://zuul.opendev.org/t/openstack/build/323afb9d5fd94f62b0bba4bac6004442/log/logs/grenade.sh.txt.gz#13753
14:18:45 kashyap Is a rebase really necessary here? - https://review.opendev.org/#/c/695056/
14:20:48 sean-k-mooney gmann: intersting
14:25:29 johnsom I wonder if it is using the py2 python-openstackclient
14:26:24 efried johnsom: I had thought so too, but similar commands just above that seem to be working fine
14:26:58 johnsom I see most of OSC also installed in the py2 environment there.
14:27:35 efried ohhh
14:27:47 efried the working commands above that aren't mucking with providers
14:27:59 efried so yeah, it's probably a matter of osc-placement being installed on the wrong py version.
14:28:19 efried gmann: ^
14:28:49 sean-k-mooney it should be python 3 https://zuul.opendev.org/t/openstack/build/b26bd7fc82d94d4cac97b5481623b629/log/logs/grenade.sh.txt.gz#16522
14:29:02 sean-k-mooney but if it was install on python 2 first
14:29:06 johnsom Well, it looks like it was properly installed in the 3.6 environment given it was a python3 devstack setup. It's just somehow the script is using the py2 openstack command.
14:29:10 sean-k-mooney then the console script would be python 2
14:29:21 johnsom https://zuul.opendev.org/t/openstack/build/323afb9d5fd94f62b0bba4bac6004442/log/logs/pip2-freeze.txt.gz#77
14:30:19 sean-k-mooney oh i know what the issue is
14:30:34 sean-k-mooney the first run install in python 2 right?
14:30:35 gmann yeah, let's recheck as devstack is all py3 now ?
14:30:39 johnsom Well, I guess that explains why it only fails when devstack is set to python3
14:30:48 sean-k-mooney e.g. train would run with python 2
14:30:56 gmann oh yeah
14:31:00 sean-k-mooney then we upgrade to ussuri with python 3
14:31:16 sean-k-mooney and we keep the ocs console script form python 2
14:31:27 sean-k-mooney then we install osc-placmeent in py36
14:31:36 sean-k-mooney which the python2 version wont find
14:31:43 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: Pick NEXT_MIN libvirt/QEMU versions for "V" release https://review.opendev.org/694821
14:31:44 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Ussuri" https://review.opendev.org/695056
14:35:49 sean-k-mooney so we would not see this if we set USE_PYTHON3=True in the grenade job
14:36:03 sean-k-mooney since both version would run on python 3
14:37:07 sean-k-mooney and in a real install the package manager/installer would unistall the python 2 versions when installing the python3 version for ussuri
14:37:40 sean-k-mooney or in container land you would jsut spin up the new python3 only container inplace of the old contianer
14:38:13 sean-k-mooney im pretty sure kolla already move to python 3 contaienr in train for what its worth
14:41:24 openstackgerrit Matt Riedemann proposed openstack/nova master: PoC for using COMPUTE_SAME_HOST_COLD_MIGRATE https://review.opendev.org/695220
14:48:00 mriedem does any of this grenade talk have anything to do with nova/
14:48:02 mriedem ?
14:48:20 mriedem still the resource provider thing or what?
14:52:43 johnsom lol, well, it's the nova scripting in grenade that is failing. That is the tie back to nova.
14:54:08 mriedem i could have sworn at one point in grenade we had some kind of hack where we'd re-install python-openstackclient b/c we went from 2 to 3
14:54:10 mriedem but i'm not finding that
15:15:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Avoid spurious error logging in _get_compute_nodes_in_db https://review.opendev.org/695453
15:17:35 ayoung What does Nova do to modify the kernel command line? It has to happen prior to cloud-init. I can see docs that imply I should be able to do this: glance image-update --property kernel_extra_args="coreos.inst.ignition_url=http://example.com/config.ign " coreos-xx-bootstrap
15:17:59 ayoung but that does not show up when the image boots, and I am guessing I need to pass that through to the instance somehow
15:19:13 sean-k-mooney if you pass a seperate kernel image in addtion to the root image i think we can pass the kernel arges to qemu
15:19:34 sean-k-mooney but in general im not sure how much that featuer is used or tested
15:19:57 sean-k-mooney i do not belive you can use it with just a root image
15:20:10 johnthetubaguy ayoung did you try os_command_line: https://github.com/openstack/nova/blob/1cd5563f2dd2b218db2422397c8aab394d484626/nova/objects/image_meta.py#L463
15:20:43 johnthetubaguy hmm, I am not so sure we do anything with that, ignore me
15:20:44 sean-k-mooney johnthetubaguy: isnt os_command_line jsut for lxc and other containers
15:20:57 sean-k-mooney like openvz
15:21:10 mriedem "The kernel command line to be used by the libvirt driver, instead of the default. For Linux Containers (LXC), the value is used as arguments for initialization. This key is valid only for Amazon kernel, ramdisk, or machine images (aki, ari, or ami)."
15:21:24 johnthetubaguy yeah, my bad

Earlier   Later