| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-13 | |||
| 16:28:04 | mriedem | i just don't know if it could throw anyone off, i.e. upgrade reno or anything, | |
| 16:28:11 | mriedem | but if you're doing upgrade_levels=auto, | |
| 16:28:19 | mriedem | you're opting into the system figuring this out | |
| 16:28:25 | mriedem | and can pin manually if you don't trust the system | |
| 16:28:42 | mriedem | let's pretend i never mentioned this | |
| 16:28:59 | dansmith | yep | |
| 16:35:22 | cdent | mriedem: I've added some links to https://bugs.launchpad.net/openstack-gate/+bug/1808063 for the recent failure that you rechecked. g-api is having a stackoverflow | |
| 16:35:22 | openstack | Launchpad bug 1808063 in OpenStack-Gate "glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server snapshot" [Undecided,Confirmed] | |
| 16:38:45 | mriedem | oh nice | |
| 16:38:46 | mriedem | http://logs.openstack.org/41/617941/31/gate/grenade-py3/7909f30/logs/screen-g-api.txt.gz#_Dec_13_14_43_21_414511 | |
| 16:38:50 | mriedem | Dec 13 14:43:21.414511 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 926 in _heartbeat_thread_job | |
| 16:39:03 | mriedem | dansmith: look familiar? ^ | |
| 16:39:10 | cdent | and a second comment | |
| 16:39:25 | mriedem | File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__ looks like infinite loop? | |
| 16:39:33 | cdent | eyes | |
| 16:39:35 | cdent | yes | |
| 16:40:03 | mriedem | https://github.com/openstack/glance/commit/cb45edf5c81f7d09c9ef0b88d40d56b4750beb10#diff-5afd3797ee6a482f2c0d4862140f7014 was the last change in there but awhile ago | |
| 16:40:42 | mriedem | https://github.com/openstack/glance/blob/master/glance/quota/__init__.py#L168 | |
| 16:44:54 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Only warn about not having computes nodes once in rpcapi https://review.openstack.org/625051 | |
| 17:02:12 | melwitt | mriedem: +1 to flushing queens and pike backports and releasing before the holiday break. I assumed not rocky bc we did that semi recently but were you thinking rocky too? I'm sure a lot has landed since last time | |
| 17:02:37 | melwitt | I will help flush | |
| 17:11:22 | mriedem | melwitt: yes otherwise we could release things in pike/queens that are not yet released in rocky | |
| 17:11:31 | mriedem | generally need to snowball these from newest to oldest branch | |
| 17:11:44 | melwitt | good point, I should have thought of that. thanks | |
| 17:12:07 | openstackgerrit | Merged openstack/nova stable/rocky: Handle tags in _bury_in_cell0 https://review.openstack.org/624813 | |
| 17:14:31 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Make compute rpcapi version calculation check all cells https://review.openstack.org/625060 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: Propagate exception message from _prep_block_device https://review.openstack.org/624832 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: Add method to generate device names universally https://review.openstack.org/573066 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 17:40:47 | colby_ | Hey Eveyrone. We just upgraded to queeens. Im seeing some weird issues when creating an instance from an already created volume. Im seeing this error: | |
| 17:40:47 | colby_ | Build of instance aborted: Nova does not support Cinder API version 2. The catalog is set to use v3 cinder as default, and in the nova config we have this set for cinder: catalog_info=volumev3:cinderv3:internalURL | |
| 17:41:06 | colby_ | we have a volumev3 in the catalog as well that points to the v3 api | |
| 17:55:20 | melwitt | colby_: is it possible you have [cinder]endpoint_template set, which will override catalog_info? | |
| 18:01:55 | cezary_zukowski | Hey, how to simulate/mimic a network disruption occuring between a Nova instance/server and Cinder (volume/storage)? Does detaching a volume from instance manually would be a sensible approach/substitute? | |
| 18:01:57 | mriedem | that would be my guess, endpoint_template is set | |
| 18:02:20 | cezary_zukowski | Command for detaching: nova volume-detach <instance_id> <volume_id> | |
| 18:02:48 | cezary_zukowski | mriedem: What do you mean? | |
| 18:03:38 | colby_ | melwitt: no its not set in the config anywhere | |
| 18:03:46 | mriedem | cezary_zukowski: was answering a different question | |
| 18:03:56 | cezary_zukowski | mriedem: ah, right | |
| 18:04:29 | mriedem | colby_: do you have this? https://github.com/openstack/nova/commit/c7d87f6691bbc39daca918729b769cc2d2000abd | |
| 18:04:45 | mriedem | that's in 17.0.6 | |
| 18:04:50 | mriedem | if you're queens ga you wouldn't have that | |
| 18:05:13 | mriedem | not sure if that's your problem though | |
| 18:17:42 | melwitt | colby_: aside from all of that, I would just take a look at your keystone catalog again and double check there's no /v2.0 for nova (should be /v2.1) and no /v2 for cinder | |
| 18:18:33 | melwitt | because the only other thing I see the code doing to determine the cinder version is parsing the version from the endpoint found in the catalog | |
| 18:18:45 | colby_ | oh it has the wrong config on the actual compute nodes. Thanks that helped me narrow it down. Ill switch that config there. Apparently it just got switched on the controller services | |
| 18:20:14 | melwitt | ok, cool | |
| 18:45:35 | mpiwowarczy | hello | |
| 18:46:36 | mpiwowarczy | can I have a question to nova deprecated personality parameter? | |
| 18:47:14 | mpiwowarczy | seems it will be removed in microversion 2.57 | |
| 18:47:42 | mpiwowarczy | but can not found when 2.57 will be used? | |
| 18:47:55 | mpiwowarczy | for Rocky release it was microversion 2.27 | |
| 18:48:06 | mpiwowarczy | which version will be used in Stein? | |
| 18:49:01 | mpiwowarczy | or is there any fixed date when 2.57 version will be dropped? | |
| 18:54:22 | artom | mpiwowarczy, microversions are never dropped, we still support 2.1 if the client asks for it | |
| 18:54:34 | artom | mpiwowarczy, in terms of releases, have a look at https://docs.openstack.org/nova/latest/reference/api-microversion-history.html | |
| 18:56:49 | mpiwowarczy | artom: thanks for the answer | |
| 18:56:56 | mpiwowarczy | so when personality files (files injection) feature will be dropped? | |
| 18:58:11 | mpiwowarczy | will be this supported in Stein release? | |
| 19:01:35 | artom | mpiwowarczy, it'll never be dropped per se | |
| 19:02:05 | artom | mpiwowarczy, you can keep using it in Stein if the client asks for a microversion < 2.57 | |
| 19:03:49 | mpiwowarczy | ok, thanks for your help | |
| 19:07:37 | melwitt | mpiwowarczy: it's dropped in a sense that if/when you find yourself needing to request microversion >= 2.57 when you call the REST API to get a different feature/addition, you will not be able to use personality files at the same time/in the same call | |
| 19:08:40 | mpiwowarczy | melwitt: I see | |
| 19:10:15 | mpiwowarczy | do you intend to maintain all versions, starting from 2.1, in all future OpenStack releases? | |
| 19:11:22 | mpiwowarczy | day by day it is more difficult to maintain :) | |
| 19:18:33 | mriedem | what is more difficult to maintain? writing apps against the API? or maintaining the various microversions in the API? | |
| 19:18:54 | mriedem | the client opts into the behavior, so if you are ok with v2.1 then you don't need to ever change that | |
| 19:20:01 | mpiwowarczy | oh yeah, you're right | |
| 19:20:49 | mpiwowarczy | just one more question - latest novaclient 11.1.0, which microversion it uses? | |
| 19:20:59 | mriedem | the alternative is we don't use microversions, change the api from release to release, and all client software everywhere in the world has to deal with that based on the cloud they are talking to | |
| 19:21:00 | mpiwowarczy | where I can check that? | |
| 19:21:27 | mriedem | the client-side support is defined here https://github.com/openstack/python-novaclient/blob/master/novaclient/__init__.py#L28 | |
| 19:21:35 | mriedem | novaclient python APIs by default use 2.1 | |
| 19:21:51 | mriedem | the CLI by default negotiates the highest supported microversion between the client and the server | |
| 19:22:24 | mriedem | so if you're using novaclient as a python sdk, then it's opt-in, | |
| 19:22:37 | mriedem | if you're using the CLI, it's negotiated unless you tell it otherwise | |
| 19:23:09 | mpiwowarczy | it is about SDK | |
| 19:23:14 | mriedem | then it's opt-in | |
| 19:23:27 | mpiwowarczy | we have defined version python-novaclient==11.1.0 and we use file injection | |
| 19:23:32 | mriedem | api methods in novaclient are wrapped / bound based on what versions they support | |
| 19:23:49 | mriedem | as i said, unless you opt into a newer microversion, it will use 2.1 | |
| 19:23:51 | mriedem | for the sdk | |
| 19:23:55 | openstackgerrit | Merged openstack/nova stable/rocky: Harden placement init under wsgi https://review.openstack.org/617297 | |
| 19:24:11 | mpiwowarczy | *python-novaclient>=9.1.0 # Apache-2.0 | |
| 19:25:05 | mpiwowarczy | thanks, you helped me a lot | |
| 19:25:32 | mriedem | yw | |
| 19:33:26 | melwitt | mriedem: do happen to know if we have the opposite of --availability-zone ZONE:HOST:NODE for admins where it would mean launch the server anywhere but this node? curious because I know in the request spec we have both force_hosts and ignore_hosts | |
| 19:33:50 | melwitt | I didn't think we did but not 100% sure | |
| 19:34:13 | mriedem | not from the api i don't think so | |
| 19:34:24 | mriedem | ignore_hosts is for when you're moving a server and you ignore the source host from schedluing | |
| 19:34:27 | cdent | i've had people ask about that too, and also couldn't find it | |
| 19:34:30 | mriedem | or ignore hosts you've rescheduled from | |
| 19:35:01 | melwitt | thanks. someone asked recently and I told them I think forbidden traits is the closest thing to something like that | |
| 19:35:21 | mriedem | well, | |
| 19:35:45 | mriedem | for that to work, you'd have to temporarily put a custom trait on the blacklisted host, schedule to !host via forbidden trait, and then remove the trait | |
| 19:36:03 | mriedem | if you're going to do that, you might as well just disable the compute service and then re-enable it after the scheduling is done | |
| 19:36:11 | mriedem | b/c the ComputeFilter essentially does the same thing | |
| 19:36:19 | mriedem | unless you're cern and you drop the max placement results down to 10 | |