Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-10
20:10:38 dansmith imacdonn: right
20:10:38 openstack Launchpad bug 1795982 in OpenStack Compute (nova) "/os-console-auth-tokens/{console_token} API doesn't handle the database backend" [High,In progress] - Assigned to melanie witt (melwitt)
20:10:38 mriedem melwitt: isn't this more than just the db backend for console auth? https://bugs.launchpad.net/nova/+bug/1795982 - it's been regressed since multi-cell support in pike
20:10:56 dansmith imacdonn: maybe just tweak and push that up and we can argue about it after there's something to see?
20:11:12 imacdonn dansmith: roger. will do after lunch
20:11:12 dansmith I'm getting kinda frustrated with this overly minute detail and really want to get on to other stuff before I end my day
20:11:16 dansmith imacdonn: thanks
20:11:30 melwitt mriedem: I don't think so because nova-consoleauth is global, you don't need to know anything about cells to query it
20:11:34 sean-k-mooney dansmith: yes i was referring to "total_run==0 and exceptions" for retrun 2
20:12:10 efried dansmith, imacdonn: I don't intend to get super involved in this, but have we considered allowing behavior change(s) (e.g. where exceptions cause failure) based on an env var or CLI switch? (Sorry, this thought has been bouncing around in my head for a week, had to get it out.)
20:12:10 melwitt mriedem: that is, it's storage of console token auths is global across all cells
20:12:40 dansmith efried: I don't think we need a behavior change here
20:12:48 mriedem hmm, ok https://docs.openstack.org/nova/queens/user/cellsv2-layout.html#consoleauth-service-and-console-proxies
20:12:52 efried ...so that existing automations aren't affected, but new users can have the benefit of potentially improved UX
20:12:54 mriedem got confused about what's global or not
20:12:57 dansmith efried: if we did, then sure
20:13:04 openstackgerrit Artom Lifshitz proposed openstack/nova master: Handle volume API failure in post_live_migration https://review.openstack.org/609517
20:13:39 sean-k-mooney dansmith: imacdonn you know what im fine with what ever you implement and ill read teh code after to understand what the exit code actully mean.
20:13:51 efried Okay. When I stopped looking, the patch was suggesting a behavior change that seemed fair and just to me, but was nacked for that reason.
20:14:57 imacdonn efried: we've introduced a new exit status, so the logic for the existing ones doesn't have to change .. I think it'll only "break" when there are exceptions and no more possible migrations, and we *want* it to break in that case
20:15:02 dansmith efried: the only thing that could make this more dreadful would be two behaviors in the same set of code :)
20:15:13 dansmith imacdonn: ++
20:15:40 efried dansmith: Hm, we should introduce a microversioning system for that.
20:15:54 dansmith efried: we should microversion your butt.
20:16:22 dansmith if we ever have nova-manage cellv2 thingy --cli-version=2.123
20:16:25 dansmith then just shoot me
20:16:54 melwitt nova-consoleauth service
20:16:54 melwitt mriedem: yeah, so the /os-console-auth-tokens/{console_token} API calls nova-consoleauth over RPC, and nova-consoleauth was made cell-aware sometime in the past, so all was working fine with multi-cell (unless there's a bug we don't know about). but when we moved to the database backend, that's what made it so that the /os-console-auth-tokens API would need to be able to talk to cell databases directly instead of going through the
20:17:37 efried what we really need is nested microversions
20:17:59 sean-k-mooney imacdonn: retruaning any new code will still requrie autoation scirpt to handel the new error case or manual intervention so the fact there is a behavior change or not is slight less important that said in the sucess case the codes should not change
20:18:01 mriedem god i bet my cross-cell resize stuff needs to recreate console auth tokens for the moved instance in the target cell db too...
20:18:03 mriedem like bdms and tags
20:18:13 mriedem and virtual_interfaces...
20:19:18 melwitt you could probably just punt that though, let them have to get a fresh console token after a cross-cell move
20:19:31 imacdonn sean-k-mooney: that's true, and that's why there's a release note for this, and it may not be backportable ... but the most common case will be running the command without --max-count and expecting a 0, or rerunning it until you don't get 1
20:19:55 melwitt default TTL for console token auth is 10 minutes so they aren't designed to live long. operators can configure longer TTL but I'm not sure they'd expect you to solve for that
20:20:44 imacdonn sean-k-mooney: I guess the possible case where it could break automation is if the command is being rerun infinitely until it gets a zero, which would never happen if it's returning 2 every time
20:20:47 sean-k-mooney imacdonn: if i was writingin this in ansiable and i called it without --max-count i would have interpereted a non 0 result as an error just fyi
20:21:14 imacdonn sean-k-mooney: right, and we're not going to change that
20:21:45 imacdonn sean-k-mooney: without --max-count, you'll either get 0 (it worked), or 2 (something unexpectedly broke, and you need to figure out why)
20:22:42 sean-k-mooney imacdonn: if you gurarentee that in the code that is fine
20:23:15 imacdonn sean-k-mooney: I believe I can .. I'll post it this afternoon, and we can nit-pick :)
20:24:22 sean-k-mooney ill be offline by then but enjoy your lunch and ill take a look at it tomorow
20:24:33 imacdonn k, thanks!
20:34:46 sean-k-mooney have people see a 404 failing to retriva allocationf form resource provierded before
20:34:48 sean-k-mooney http://logs.openstack.org/84/602384/4/check/kuryr-kubernetes-tempest-daemon-octavia/33adb32/controller/logs/screen-n-cpu.txt.gz?#_Oct_10_19_02_09_953771
20:36:36 openstackgerrit Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
20:36:49 openstackgerrit Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
20:56:04 mriedem melwitt: ok +2
20:57:47 melwitt thanks
21:10:12 mriedem sean-k-mooney: yes it's a known bug
21:10:41 openstack Launchpad bug 1789998 in OpenStack Compute (nova) "ResourceProviderAllocationRetrievalFailed ERROR log message on fresh n-cpu startup" [Low,Triaged]
21:10:41 mriedem https://bugs.launchpad.net/nova/+bug/1789998
21:10:50 mriedem happens on every start of a new compute
21:14:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't log an error if attachment_create fails https://review.openstack.org/609547
21:23:03 sorrison mriedem: Missing some context RE: "maybe that's just never been a problem b/c of quota restrictions, the multi-create thing i mean"
21:23:30 mriedem meaning maybe no one has ever felt the need to restrict certain groups of users from being able to make multi-create requests
21:24:00 mriedem because multi-create can be abused, e.g. https://review.openstack.org/#/c/607735/
21:24:41 mriedem sorrison: maybe a better question is, what is the highest any of your tenants have for instance quota?
21:26:46 sorrison mriedem: 2048 is the highest just looking in our DB
21:26:52 mriedem jesus
21:27:03 mriedem have you ever tried to create 2048 servers in a single create request?
21:27:14 sorrison haha don't be silly :-)
21:27:17 mriedem because the API will let you do that
21:27:34 mriedem there is no rate limiting on multi-create requests
21:27:46 mriedem and that size of request will melt your scheduler
21:28:03 mriedem related: https://review.openstack.org/#/c/510235/
21:28:08 sorrison is the multi create number set in the request spec? I can have a look in the db to see what our stats are like
21:28:38 mriedem yes, it's the request spec "num_instances" field
21:35:39 openstackgerrit Merged openstack/os-vif master: clean up ip_command interface https://review.openstack.org/609414
21:36:32 sorrison mriedem: very tricky to get that info out of mysql due to json blob. We're not running a version of mysql that has json support sadly
21:36:51 mriedem damn
21:40:38 melwitt I never knew request spec was a json blob until now O.o
21:40:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't log error in _remove_deleted_instances_allocations if compute is new https://review.openstack.org/609552
21:41:18 sorrison either way, we're aware of the issue but haven't had any major issues
21:41:42 mriedem maybe i can get some data from some public cloud ops
21:41:46 sorrison Just trying to get my sql foo on to see if I can extract the num_instances
21:51:46 sorrison ok so max num_instances we've had is 49
21:52:48 sorrison mriedem: stats here http://paste.openstack.org/show/731867/
21:53:23 melwitt neat
21:56:53 mriedem sorrison: nice, thanks
22:01:13 sorrison mriedem: I sorted out https://review.openstack.org/#/c/608474/ still not sure about correct name for policy as it does affect list and show for a flavor
22:01:55 mriedem that's not checked on show is it?
22:02:05 mriedem or you mean, allow non-admins to show private flavors that they don't have access to?
22:02:21 mriedem i.e. support person trying to triage a bug for a server created with a private flavor?
22:07:40 sorrison Yes
22:07:55 sorrison The change to allow policy for a flavor show is https://review.openstack.org/#/c/608474/3/nova/objects/flavor.py
22:08:10 sorrison it's for reporting scripts
22:09:16 mriedem yup took me a second to sort that out
22:09:31 sorrison yeah it goes down a few layers from the api
22:12:29 mriedem sorrison: ok comments inline;
22:12:41 mriedem i left some suggestions about the rule name, but they aren't awesome
22:12:49 mriedem maybe alex_xu or gmann or dansmith have ideas
22:15:06 sorrison yeah I can't think of something good there
22:16:04 sorrison with disabled flavors I think they are useful. We are planning on retiring some flavors soon and were planning on disabling them by updating the DB
22:16:27 sorrison unless there is a better way to retire flavors?
22:27:33 mriedem there is no way to disable flavors via the API, which is why i'm sort of hesitant to mention them
22:27:49 mriedem but it's not a big deal to leave that in if you're hacking flavors.disabled via the db directly
22:27:55 mriedem it just sucks you have to do that...
22:28:09 mriedem we have a PUT /flavors/{flavor_id} now...seems that would be a place to disable/enable flavors

Earlier   Later