Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-19
14:59:58 dansmith bauzas: yeah
15:00:37 bauzas dansmith: ack, just wondering why https://review.opendev.org/#/c/761452/ is getting the 6.0 version as the minor version
15:00:53 bauzas " Nov 13 10:14:02.664855 ubuntu-focal-rax-iad-0021755641 devstack@n-api.service[72892]: ERROR nova.api.openstack.wsgi [None req-dbe8b15b-bd32-46ab-93b4-b0097a6a86a3 None None] Unexpected exception in API method: oslo_messaging.rpc.client.RPCVersionCapError: Requested message version, 5.0 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 6.0."
15:01:22 lpetrut sean-k-mooney: so basically each guest can update its own metadata by just sending it to http://169.254.169.254/openstack
15:01:51 sean-k-mooney lpetrut: that might be a bug. i think that was only ment to be updated via the nova api
15:01:57 bauzas dansmith: but I provided an additional endpoint for the v5.0 proxy
15:03:10 dansmith bauzas: well, you haven't implemented 6.0 in the client, but all the servers are reporting that they support 6.0 (via the service version) and everything is new, so it's choosing to use 6.0 but the client doesn't support it, right?
15:04:11 lpetrut sean-k-mooney: fwiw here's where cloudbase-init is sending the password: https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/metadata/services/httpservice.py#L55-L80
15:04:31 dansmith bauzas: remember, the client has to be able to speak both versions as well, depending on the version pin, but it's choosing to speak the new one based on the service version, and you only speak 5.x on the client side right now
15:04:41 bauzas dansmith: sure, but if the client is only supporting 5.0, the server should still accept it given the additional endpoint, nope ?
15:04:56 lpetrut sean-k-mooney: I wouldn't say it's a bug, it allows the guest to communicate back various metadata items. I'm not sure if it's filtered in any way.
15:04:57 dansmith bauzas: yes, but the auto pin will try to configure the client with 6.0
15:04:59 sean-k-mooney lpetrut: i see well that is undocumented behavior that they are relying on i think
15:05:10 dansmith bauzas: because everything is new
15:05:30 bauzas dansmith: why this worked then with https://review.opendev.org/#/c/541005/6 ?
15:06:02 sean-k-mooney lpetrut: unless im mistaken and you can point me to docs or a spec to the controy i dont think that was ever intended to work
15:06:21 dansmith bauzas: were we auto calculating the pin then? we'd have to look at devstack config from back then to know
15:06:29 lpetrut sean-k-mooney: I'm sure I can find some docs on this, checking
15:07:01 bauzas dansmith: maybe, I dunno
15:09:43 sean-k-mooney lpetrut: this is the metadata docs https://docs.openstack.org/nova/latest/user/metadata.html
15:10:05 dansmith bauzas: Nov 13 10:14:02.660873 ubuntu-focal-rax-iad-0021755641 devstack@n-api.service[72892]: INFO nova.compute.rpcapi [None req-dbe8b15b-bd32-46ab-93b4-b0097a6a86a3 None None] Automatically selected compute RPC version 6.0 from minimum service version 54
15:10:18 dansmith bauzas: that's trying to configure rpcapi.py with version 6.0, but it doesn't support that
15:11:29 bauzas dansmith: yup, I understood it
15:11:46 bauzas dansmith: OK, I can try to support 6.0 for the rpcapi
15:12:03 sean-k-mooney lpetrut: that behavior is not descitbe in either the metadata docs or api docs
15:12:10 dansmith bauzas: well, you have to :)
15:12:25 sean-k-mooney lpetrut: when you do that is the metadata stored back to the db
15:12:35 sean-k-mooney they way it would be if you called the nova api
15:12:38 lpetrut sean-k-mooney: fwiw, here's an explicit handler for "POST": https://github.com/openstack/nova/blob/master/nova/api/metadata/password.py#L62
15:12:54 sean-k-mooney lpetrut: right that is for the nova api
15:13:06 sean-k-mooney it supproted if you call the nova api with an authenicated token
15:13:21 sean-k-mooney but doing it form the guest vai the 169 adress is not as far as i know
15:14:22 lpetrut nope, that specific request doesn't require a token
15:14:41 lpetrut https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L214-L225
15:14:49 sean-k-mooney so this is something ill have to bring up with the security team
15:15:07 sean-k-mooney we have discussed it publicly at this point but its potentially an issue
15:15:10 lpetrut from what I can tell, the guest can only update specific fields, the password being one of them
15:16:38 lpetrut I've just checked, so actually the "password" field is the only one that can be updated
15:16:38 f0o sean-k-mooney: actually it seems that /password accepts POST on the metadata service... The issue was that the sshkey used was ECDSA and cloudbase-init went bananas. I get the password posted correctly if I load it up with an RSA key
15:17:16 sean-k-mooney f0o: ecdsa need a newer version of pycyrptography to work
15:17:21 lpetrut tbh I'm a bit surprised that the nova metadata docs don't mention this, this feature has been there since forever (<2014 I think)
15:17:48 sean-k-mooney as i said im not sure it has been
15:17:55 sean-k-mooney i think this was unitnetional
15:18:01 sean-k-mooney there is no spec for it
15:18:05 lpetrut here it is: https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89
15:18:08 sean-k-mooney i checked
15:18:15 lpetrut 2012, pre nova specs era :)
15:19:03 sean-k-mooney this is only nova v1 api behavior
15:19:30 sean-k-mooney actully no its liberty which is not pre sepecs
15:20:01 lpetrut I guess it seemed simple enough not to mandate a spec, but it's definitely intentional
15:20:23 lpetrut and it's not just nova v1
15:20:42 sean-k-mooney lpetrut: we require specs for all api cahnge regradless of how trivial it is
15:21:20 sean-k-mooney lpetrut: yes the nova v1 comment was because you said it was pre specs
15:21:32 sean-k-mooney we have specs for similar feature in libvirt https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-set-admin-password.html
15:21:37 openstackgerrit Stephen Finucane proposed openstack/nova stable/train: Add missing exception https://review.opendev.org/763393
15:21:40 lpetrut oh, got it
15:22:54 lpetrut I think that libvirt driver feature is slightly different than https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89
15:23:29 sean-k-mooney we are missing an api microverion bump for this too
15:23:52 lpetrut bump the microversion for what?
15:23:59 sean-k-mooney the metadata api
15:24:08 lpetrut https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 this is there since 2012 :)
15:24:30 lpetrut it's not something that was added now
15:24:41 sean-k-mooney yep and it was not documented and did not follow our spec proceducer and did not do an api microverion bump to the metadata api
15:25:08 sean-k-mooney lpetrut: sure imjust not sure it shoudl continue to be supported unless we at least fix the docs
15:25:40 lpetrut well, just because it wasn't properly documented doesn't mean that it should become unsupported, there are projects relying on it
15:25:41 sean-k-mooney if its just this field it might be ok. if other fiels can be arbitrally updated then it could be a an issue
15:26:03 lpetrut sean-k-mooney: indeed. well, it's ok, it's just this field
15:26:15 sean-k-mooney lpetrut: not that im aware of other the cloudbase-init
15:27:24 lpetrut well, since virtually all Windows Openstack instances use it, I'd say breaking it wouldn't be desired, even though it's just one project
15:27:25 sean-k-mooney apparently it was for hyperv https://blueprints.launchpad.net/nova/+spec/hyper-v-metadata-password-post
15:27:36 sean-k-mooney but it was not appoved
15:28:09 sean-k-mooney thre is no https://blueprints.launchpad.net/nova/+spec/get-password blueprit
15:28:18 sean-k-mooney oh there is
15:28:22 sean-k-mooney it did not come up
15:28:26 lpetrut https://blueprints.launchpad.net/nova/+spec/hyper-v-metadata-password-post seems like an extension of this feature
15:29:20 sean-k-mooney this was first added as an api extention https://review.opendev.org/#/c/17273/
15:29:41 sean-k-mooney before we removed those
15:30:36 sean-k-mooney so this was nota catully part of the metadata api
15:30:52 sean-k-mooney it was a vendor extntion https://review.opendev.org/#/c/17273/15/nova/api/openstack/compute/contrib/server_password.py
15:30:55 lpetrut nice. thanks for checking. this was an interesting lesson of Nova history :)
15:31:25 sean-k-mooney so ya i think we moved it into the metrada service wehn we got rid fo extensions
15:33:31 lpetrut sorry for mentioning the server actions, that's completely unrelated. it took a while since I last had contact with this code so I was a bit confused.
15:36:52 sean-k-mooney so this changed in liberty
15:37:05 sean-k-mooney that is when we remvod the contib folder
15:38:04 sean-k-mooney ah it move to legacy_v2 contrib
15:41:03 sean-k-mooney https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/api-no-more-extensions.html
15:42:08 sean-k-mooney so this was deprecated in libvirty and removed in newton
15:43:55 lpetrut yep, API extensions were deprecated but then got included in the nova api
15:44:24 sean-k-mooney they were not just all accpeted
15:44:33 sean-k-mooney they needed to be upstreamed
15:44:52 lpetrut makes sense. well, this specific one is part of the nova tree. seems upstream to me :)
15:46:35 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: Reproducer for unpinned to pinned resize bug https://review.opendev.org/763399
15:46:44 sean-k-mooney it got moved by https://github.com/openstack/nova/commit/003c868da73d84d33fba81ee9b033b8ae321e7ab
15:46:55 artom stephenfin, sean-k-mooney, ^^ sanity check that for me pretty please? I feel like I've missed something obvious
15:47:02 artom And yet...
15:47:46 sean-k-mooney what are yo trying to check?
15:47:54 sean-k-mooney unpinned to pinned works fine
15:47:55 stephenfin artom: You need to disable the workaround option
15:47:59 sean-k-mooney or at least it used too
15:48:14 stephenfin artom: '[workarounds] disable_fallback_pcpu_query'

Earlier   Later