| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-21 | |||
| 13:42:20 | efried | i.e. don't do the cut | |
| 13:42:32 | efried | not that that would help here, because clearly the command is failing. | |
| 13:42:52 | efried | But why is the error output going to stdout rather than stderr? | |
| 13:43:01 | mriedem | of that provider list command allowed passing a --name for filtering, we could pass the local fqdn to get 1 result back... | |
| 13:43:03 | efried | Whole point of stderr is so exactly this doesn't happen, and you can see what actually went wrong. | |
| 13:43:28 | zigo | mriedem: A much nicer way using http://harelba.github.io/q/ : provider=$(openstack resource provider list --format csv | q -H -d, "SELECT uuid FROM - LIMIT 1") | |
| 13:43:30 | zigo | ;) | |
| 13:43:51 | efried | mriedem: to that point, it looks like we don't care *which* provider we're grabbing? That seems... weird. | |
| 13:43:56 | zigo | (q-text-as-data is such a nice tool...) | |
| 13:43:59 | mriedem | efried: we don't, | |
| 13:44:12 | mriedem | it's a smoke test to make sure that we can save off some inventory before upgrading and that after the upgrade it's still there | |
| 13:44:13 | gmann | pick up anything should be fine | |
| 13:44:25 | efried | okay. | |
| 13:45:31 | zigo | mriedem: Then you could do: provider=$(openstack resource provider list --format csv | q -H -d, "SELECT uuid FROM - WHERE name='something-you-want'") | |
| 13:45:35 | efried | anyway, the problem here seems to be that the openstack command -- the first thing in the pipe -- is failing, printing its error to stdout. | |
| 13:47:32 | mriedem | zigo: or i could just do provider=$(openstack resource provider list -f value -c uuid --name `hostname -f`) | |
| 13:47:44 | mriedem | and not rely on pipes and other tooling | |
| 13:47:58 | mriedem | but that provider list command doesn't support --name (yet - that's easy to add) | |
| 13:48:48 | gmann | API has the name filter ? | |
| 13:48:51 | mriedem | yes | |
| 13:49:06 | efried | one way to get output like 'openstack: $stuff' is if the openstack command doesn't exist. But that output goes to stderr like it should. | |
| 13:50:58 | sean-k-mooney | gmann: its how neutron identifies the compute node resouce provider without needing to known the compute node uuid | |
| 13:51:10 | sean-k-mooney | it looks up the RP by hostname | |
| 13:51:10 | efried | I can't find anything in the code that's joining stderr to stdout. Unless the job itself is doing that. | |
| 13:52:17 | efried | Okay, apparently I'm the only one concerned about the fact that `openstack resource provider list` is producing bogus output in gmann's case, so I must be misunderstanding what we're actually trying to solve here. /me stfu, call if you need me. | |
| 13:53:48 | gmann | sean-k-mooney: and there it is working fine? failure case of octavia grenade job on py3. | |
| 13:54:36 | sean-k-mooney | they do that in code not via osc | |
| 13:54:47 | gmann | ohk | |
| 13:59:09 | sean-k-mooney | is looking up the provider by hostname all that is breakign the job? | |
| 13:59:57 | sean-k-mooney | it would be quick to fix osc-plamcenet to support that but equally quick to just do it with curl | |
| 14:01:32 | gmann | sean-k-mooney: fixing osc might take time with release etc until octavia job can install it from source. | |
| 14:01:54 | sean-k-mooney | do we know what cause the broken pipes? | |
| 14:02:03 | gmann | no. | |
| 14:02:21 | haleyb | sean-k-mooney: would adding osc-placement to requirements in octavia fix it as well? | |
| 14:02:53 | sean-k-mooney | i think mriedem said list does not currently support --name | |
| 14:04:02 | sean-k-mooney | i was looking at the greand job logs by the way gmann haleyb do you have the link to the octavia job | |
| 14:04:33 | johnsom | haleyb Yes, that is the error output you are seeing. Installing osc-placement should fix it. | |
| 14:04:49 | gmann | sean-k-mooney: https://review.opendev.org/#/c/693486/ | |
| 14:04:50 | johnsom | https://www.irccloud.com/pastebin/PrDd7zG6/ | |
| 14:05:08 | openstackgerrit | Merged openstack/nova stable/pike: Delete instance_id_mappings record in instance_destroy https://review.opendev.org/684658 | |
| 14:05:21 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Ussuri" https://review.opendev.org/695056 | |
| 14:05:42 | sean-k-mooney | ah right yes "'resource provider list -f value' is not an openstack command" | |
| 14:05:44 | kashyap | stephenfin: ^^ Fixed the functional test (forgot to run `tox -e functional-36`, bad me) | |
| 14:05:53 | johnsom | haleyb The question is which devstack plugin has the missing requirement | |
| 14:05:55 | sean-k-mooney | is becasue osc-placement is not installed | |
| 14:05:59 | gmann | johnsom: haleyb it is parsing of command failing not about osc command is need more installation etc | |
| 14:06:03 | kashyap | stephenfin: Thanks for the earlier review :-) | |
| 14:06:53 | kashyap | Uh, seems like I need a rebase... | |
| 14:07:04 | sean-k-mooney | johnsom: i guess the ocatavia one | |
| 14:07:17 | gmann | thus command we need to adjust to get the first RP- https://github.com/openstack/grenade/blob/fad62595bff3ae55b5b428a5ea00a9a168390fd2/projects/60_nova/resources.sh#L57 | |
| 14:07:30 | sean-k-mooney | but honestly it might be better for devstack to install osc-placemetn if placement is installed | |
| 14:07:32 | johnsom | sean-k-mooney Octavia devstack plugin isn't running that command. | |
| 14:07:40 | gmann | command i mean parsing logic | |
| 14:07:51 | sean-k-mooney | oh ok | |
| 14:08:32 | johnsom | It is openstack/grenade | |
| 14:08:44 | johnsom | projects/60_nova/resources.sh | |
| 14:08:49 | johnsom | line 57 | |
| 14:08:51 | sean-k-mooney | gmann: im confused on the octaiva patch you are linink to grenade | |
| 14:09:05 | sean-k-mooney | is this a but there are other fialng jobs too | |
| 14:09:14 | sean-k-mooney | *failing jobs | |
| 14:09:27 | sean-k-mooney | are ye only looking at the grenade one at the moment | |
| 14:10:02 | gmann | yeah grenade one only i checked | |
| 14:10:39 | sean-k-mooney | well we could have grenade install it or as i said we could have devstack install it if placemnt is installed | |
| 14:11:12 | sean-k-mooney | grenade is using it so its resonable for it to install its own depencies | |
| 14:11:20 | johnsom | Yeah. It's odd that grenade doesn't have a requirements.txt though it obvious has requirements in it's scripts | |
| 14:11:46 | sean-k-mooney | well grenade is not a python porject | |
| 14:12:39 | sean-k-mooney | its almost all bash so you would not pip install it | |
| 14:13:23 | gmann | osc-placement installation is not the issue here. | |
| 14:13:26 | johnsom | Ha, yeah, I just noticed that. I guess you know now how much time I have looked at grenade.... lol | |
| 14:14:09 | johnsom | gmann Yes it is. If that is not installed the first output of OSC is "openstack:" which the script tries to parse. | |
| 14:15:31 | sean-k-mooney | right as your irccloud link show the message is "openstack: 'resource provider list -f value' is not an openstack command. See 'openstack --help'." | |
| 14:15:47 | johnsom | Yep | |
| 14:17:13 | gmann | https://zuul.opendev.org/t/openstack/build/323afb9d5fd94f62b0bba4bac6004442/log/logs/grenade.sh.txt.gz#13753 | |
| 14:18:45 | kashyap | Is a rebase really necessary here? - https://review.opendev.org/#/c/695056/ | |
| 14:20:48 | sean-k-mooney | gmann: intersting | |
| 14:25:29 | johnsom | I wonder if it is using the py2 python-openstackclient | |
| 14:26:24 | efried | johnsom: I had thought so too, but similar commands just above that seem to be working fine | |
| 14:26:58 | johnsom | I see most of OSC also installed in the py2 environment there. | |
| 14:27:35 | efried | ohhh | |
| 14:27:47 | efried | the working commands above that aren't mucking with providers | |
| 14:27:59 | efried | so yeah, it's probably a matter of osc-placement being installed on the wrong py version. | |
| 14:28:19 | efried | gmann: ^ | |
| 14:28:49 | sean-k-mooney | it should be python 3 https://zuul.opendev.org/t/openstack/build/b26bd7fc82d94d4cac97b5481623b629/log/logs/grenade.sh.txt.gz#16522 | |
| 14:29:02 | sean-k-mooney | but if it was install on python 2 first | |
| 14:29:06 | johnsom | Well, it looks like it was properly installed in the 3.6 environment given it was a python3 devstack setup. It's just somehow the script is using the py2 openstack command. | |
| 14:29:10 | sean-k-mooney | then the console script would be python 2 | |
| 14:29:21 | johnsom | https://zuul.opendev.org/t/openstack/build/323afb9d5fd94f62b0bba4bac6004442/log/logs/pip2-freeze.txt.gz#77 | |
| 14:30:19 | sean-k-mooney | oh i know what the issue is | |
| 14:30:34 | sean-k-mooney | the first run install in python 2 right? | |
| 14:30:35 | gmann | yeah, let's recheck as devstack is all py3 now ? | |
| 14:30:39 | johnsom | Well, I guess that explains why it only fails when devstack is set to python3 | |
| 14:30:48 | sean-k-mooney | e.g. train would run with python 2 | |
| 14:30:56 | gmann | oh yeah | |
| 14:31:00 | sean-k-mooney | then we upgrade to ussuri with python 3 | |
| 14:31:16 | sean-k-mooney | and we keep the ocs console script form python 2 | |
| 14:31:27 | sean-k-mooney | then we install osc-placmeent in py36 | |
| 14:31:36 | sean-k-mooney | which the python2 version wont find | |
| 14:31:43 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: Pick NEXT_MIN libvirt/QEMU versions for "V" release https://review.opendev.org/694821 | |
| 14:31:44 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Ussuri" https://review.opendev.org/695056 | |
| 14:35:49 | sean-k-mooney | so we would not see this if we set USE_PYTHON3=True in the grenade job | |