Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-29
14:05:11 ganso has seen this problem before and it is an obvious one before I open the bug. Unfortunately I don't have access to the environment so it is hard for me to collect more data, and I couldn't reproduce in my lab. So, has anyone seen this weird behavior before?
14:05:11 ganso Hi folks. I am about to open a new nova bug that my customer is experiencing. They are running "openstack usage list --start 2019-08-01 --end 2019-08-31 --os-compute-api-version 2.40" and it stays in infinite loop. With --debug, I could see that the marker is never updated, therefore the server always sends the same page result over and over. The env is Queens, this was tested with OpenStack CLI 3.19 and 4.0. I decided to check here in case someone
14:05:22 mriedem so i guess the question is, would we consider it a bug that you can generate and change the admin password on a guest without getting the value back?
14:05:32 mriedem because if that's a bug, we might not require a microversion to change it
14:05:58 mriedem efried: osc is meant to run against old server versions
14:06:07 mriedem dtroyer is pretty adament about backward compat with old clouds
14:06:12 mriedem even for like nova-network support
14:06:14 efried how much older? The real question is: when can we drop nova-network support from osc?
14:06:16 efried d'oh
14:06:28 mriedem why are you asking?
14:06:44 mriedem we dropped nova-network support from novaclient and dean had to fix that all up in osc to continue supporting nova-net
14:07:02 efried because osc generates option helps based on whether it detects the 'network' (neutron) service or not.
14:07:04 sean-k-mooney can you tell form the api that it is nova networks?
14:07:11 efried which is fine when you're running in a cloud
14:07:22 efried but not so much when you're generating html docs to publish.
14:07:58 sean-k-mooney efried: well the sphinx docs for osc should generate for all intree options right
14:08:03 mriedem ganso: never heard of it but i wouldn't be surprised with markers, though it would be good to know more details about the marker, e.g. if it's a deleted server
14:08:09 sean-k-mooney and then plugins have there own docs
14:08:21 sean-k-mooney efried: do you wnat to skip the nova networks docs?
14:08:58 mriedem efried: option help or entire commands?
14:09:01 efried both neutron and nova-net options are in osc proper, non-plugin. Yes, I would love to be able to code that decision as: "If I'm not running in a cloud, assume I'm building docs, and since nova-network is dead, only generate the neutron options"
14:09:32 efried mriedem: uhm, not totally sure about that, would have to dig a bit more.
14:09:52 mriedem i'd say either drop nova-network stuff from published docs or you'd have to mark anything as nova-net as nova-net specific i guess? which probably sucks.
14:10:57 efried right
14:10:57 ganso mriedem: hmm that is an interesting detail. So, the marker I have is exactly the last instance returned from the first request. In the dict returned by the API, the instance is marked as "active". The CLI then uses this instance's ID for the next request URL, but then the API just returns the same page, with the same instance as last one. And over and over.
14:11:00 sean-k-mooney efried: given the last time i deployed with nova net osc was not a thing i have never seen the nova net commands/opetions do you have an example
14:11:58 ganso mriedem: Even though the instance is marked as "active" in the returned API, it is possible that the instance has been deleted but was active in the queried period?
14:12:34 efried sean-k-mooney: I'm not sure whether the docs have any of those today. Which would make my job easier, as what I'm doing will just produce the equivalent. Let me go dig some...
14:12:39 efried sean-k-mooney: fyi https://review.opendev.org/#/c/691767/1/openstackclient/network/common.py@174
14:13:57 mriedem ganso: i would expect that if the instance is deleted then it's vm_state is not active
14:14:18 mriedem or you're really unlucky in hitting a tight race window
14:14:57 mriedem i just asked about deleted markers because we've seen bugs with those so i wouldn't be surprised if that is the issue
14:15:21 ganso mriedem: hmm ok so the usage returns the current instance state. About races, the customer can consistently reproduce this, it happens every time
14:15:41 mriedem do you have this fix? https://review.opendev.org/#/c/632516/
14:15:53 mriedem that is in queens 17.0.10
14:16:06 ganso mriedem: yes. Customer is running 17.0.11
14:16:31 sean-k-mooney efried: for what its worth looking at https://docs.openstack.org/python-openstackclient/pike/cli/command-list.html and drilling down it port and network resouce i dont see nova network sepcific commands
14:17:00 mriedem https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/server.html#server-ssh is nova-net specific
14:17:08 efried sean-k-mooney: one example is https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/security-group.html#security-group-list
14:17:09 efried And now I'm noticing that they're tagged.
14:17:09 efried In this case, the nova-network options (just --all-projects) are a subset of the neutron options (which is what the doc has)
14:17:22 sean-k-mooney mriedem: you can use the ssh command with neutron
14:17:44 sean-k-mooney mriedem i use it all the time when im too lazy to lookup the ip
14:19:07 sean-k-mooney efried why is that nova networks specific. it should list the neutorn security groups
14:19:24 sean-k-mooney and --all-project shoudl still work in that context if you are an admin
14:19:45 efried sean-k-mooney: I'm saying, this is an example where the same subcommand has "different" options for nova-net vs neutron.
14:19:51 efried except they overlap
14:19:57 sean-k-mooney ah right
14:20:01 efried except the current doc says --all-projects is ignored
14:20:05 sean-k-mooney Network version 2 ignores this option and will always display information for all projects (admin only).
14:20:12 efried which doesn't gel with what the opt help says
14:20:13 sean-k-mooney that is the delta
14:20:25 efried oh, yes it does
14:20:49 sean-k-mooney people might not click that "networkign v2" mean neutron v2
14:20:56 efried yeah, no kidding
14:22:18 efried okay, so nova-net only supports --all-projects, and neutron supports all the others but ignores --all-projects. (But neutron still adds the opt so it won't blow up if you use it.)
14:23:01 sean-k-mooney efried: ya that seam to be the case
14:25:16 efried okay, so to make this work the way I want, to get actual parity, I would have to produce both helps, marked with their respective support statements :(
14:25:59 dansmith ...which is what we should do if we're responsible backwards-compatible software engineers amirite?
14:26:31 efried yes -- again assuming we still want an ussuri-level osc to support nova-net
14:26:53 dansmith I think you mean s/ussuri-level//
14:28:47 efried python-openstackclient is cycle-with-intermediary. Its deliverables are aligned with openstack releases. I'm making changes to osc that will release with ussuri. So I think I mean, if it's legal to deploy an ussuri-level osc in a cloud that still runs nova-net...
14:29:28 dansmith well, I think the point is.. it's a client and I think the goal for most is to have clients be backwards compatible with stable APIs
14:30:16 efried but nova-net was only deprecated in newton. Which means there will be people who will run it until it don't run no mo. Which means stein. And surely you can run ussuri osc against a stein cloud.
14:30:43 efried So I can assume removing nova-net support from osc is a ways off in the future, and I have to deal with it.
14:31:01 sean-k-mooney efried: dansmith as a compromise assuming moveing to the sdk in ocs contindues wehn the major version bump fo that happens we could drop nova-net support as part of that
14:31:49 dansmith efried: fwiw just the other day I was talking to someone running a very large production mitaka cloud, working to deploy train for their next phase.. it would suck super bad to not have a client be able to talk to both of them
14:31:50 efried for now I don't actually care "when" or even "if"; all I needed to know was "not now".
14:31:59 efried dansmith: ack
14:33:26 efried btw, for what I'm working on now, it's not a question of dropping actual support for nova-net. It's just a question of whether docs.o.o/osc/ussuri{and later}/ will contain the options for nova-net.
14:33:51 efried but the answer is the same.
14:34:01 dansmith well, if it's in the client and not documented (even if not generated) that also kinda sucks, but less so, I'll agree
14:35:43 efried dansmith: the CLI helps only produce the content for one or the other, by detecting whether the 'network' (neutron) endpoint exists.
14:35:56 dansmith ...right
14:36:03 efried but of course we're not in your cloud when we're generating docs
14:36:07 dansmith sure
14:36:26 efried thanks for the talk, all. The path forward is clear, if messy.
14:40:30 dansmith mriedem: are you looking for me to pass all the notification payload things in via init, or only the two things you noted in the review?
14:44:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dead set_admin_password code to generate password https://review.opendev.org/691873
14:51:11 mriedem dansmith: just the few things noted
14:51:17 mriedem the image lists are generated after the fact
14:51:52 dansmith mriedem: okay I'm not sure I see the distinction, but that's fine
14:53:25 mriedem it was a nit, ignore if you want
14:54:28 dansmith it's already done, I just wanted to know the convention for next time, but if it's just "feels right" then that's fine too
15:03:18 mriedem like the spanx i'm wearing right now, it just feels right
15:04:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dead set_admin_password code to generate password https://review.opendev.org/691873
15:04:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix changePassword adminPass="" broken schema validation https://review.opendev.org/691880
15:06:30 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix changePassword adminPass="" broken schema validation https://review.opendev.org/691880
15:09:33 openstack Launchpad bug 1850153 in OpenStack Compute (nova) "nova resize operation doesn't support disk resize for emphemeral disk and swap disk" [Undecided,New]
15:09:33 mriedem oh god how many duplicates do we have of this bug https://bugs.launchpad.net/nova/+bug/1850153
15:09:36 efried mriedem: so passing in an empty password does *not* generate?
15:09:42 mriedem efried: nope
15:09:47 mriedem and you can't pass adminPass=None
15:09:50 efried passing in null theoretically would have, but that gets kicked by schema... yeah.
15:10:15 efried adminPass="" will actually set the password to the empty string.
15:10:21 sean-k-mooney yep
15:10:37 sean-k-mooney peopel try to cheat schema validation on the neutron side setting ""
15:10:42 mriedem which is probably a security vulnerability
15:10:59 sean-k-mooney to pass null and end up setting it to the litral empty sting
15:11:17 mriedem as noted in the commit, the changePassword API originally did not allow ""

Earlier   Later