Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-10
20:15:02 dansmith efried: the only thing that could make this more dreadful would be two behaviors in the same set of code :)
20:15:13 dansmith imacdonn: ++
20:15:40 efried dansmith: Hm, we should introduce a microversioning system for that.
20:15:54 dansmith efried: we should microversion your butt.
20:16:22 dansmith if we ever have nova-manage cellv2 thingy --cli-version=2.123
20:16:25 dansmith then just shoot me
20:16:54 melwitt nova-consoleauth service
20:16:54 melwitt mriedem: yeah, so the /os-console-auth-tokens/{console_token} API calls nova-consoleauth over RPC, and nova-consoleauth was made cell-aware sometime in the past, so all was working fine with multi-cell (unless there's a bug we don't know about). but when we moved to the database backend, that's what made it so that the /os-console-auth-tokens API would need to be able to talk to cell databases directly instead of going through the
20:17:37 efried what we really need is nested microversions
20:17:59 sean-k-mooney imacdonn: retruaning any new code will still requrie autoation scirpt to handel the new error case or manual intervention so the fact there is a behavior change or not is slight less important that said in the sucess case the codes should not change
20:18:01 mriedem god i bet my cross-cell resize stuff needs to recreate console auth tokens for the moved instance in the target cell db too...
20:18:03 mriedem like bdms and tags
20:18:13 mriedem and virtual_interfaces...
20:19:18 melwitt you could probably just punt that though, let them have to get a fresh console token after a cross-cell move
20:19:31 imacdonn sean-k-mooney: that's true, and that's why there's a release note for this, and it may not be backportable ... but the most common case will be running the command without --max-count and expecting a 0, or rerunning it until you don't get 1
20:19:55 melwitt default TTL for console token auth is 10 minutes so they aren't designed to live long. operators can configure longer TTL but I'm not sure they'd expect you to solve for that
20:20:44 imacdonn sean-k-mooney: I guess the possible case where it could break automation is if the command is being rerun infinitely until it gets a zero, which would never happen if it's returning 2 every time
20:20:47 sean-k-mooney imacdonn: if i was writingin this in ansiable and i called it without --max-count i would have interpereted a non 0 result as an error just fyi
20:21:14 imacdonn sean-k-mooney: right, and we're not going to change that
20:21:45 imacdonn sean-k-mooney: without --max-count, you'll either get 0 (it worked), or 2 (something unexpectedly broke, and you need to figure out why)
20:22:42 sean-k-mooney imacdonn: if you gurarentee that in the code that is fine
20:23:15 imacdonn sean-k-mooney: I believe I can .. I'll post it this afternoon, and we can nit-pick :)
20:24:22 sean-k-mooney ill be offline by then but enjoy your lunch and ill take a look at it tomorow
20:24:33 imacdonn k, thanks!
20:34:46 sean-k-mooney have people see a 404 failing to retriva allocationf form resource provierded before
20:34:48 sean-k-mooney http://logs.openstack.org/84/602384/4/check/kuryr-kubernetes-tempest-daemon-octavia/33adb32/controller/logs/screen-n-cpu.txt.gz?#_Oct_10_19_02_09_953771
20:36:36 openstackgerrit Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
20:36:49 openstackgerrit Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
20:56:04 mriedem melwitt: ok +2
20:57:47 melwitt thanks
21:10:12 mriedem sean-k-mooney: yes it's a known bug
21:10:41 openstack Launchpad bug 1789998 in OpenStack Compute (nova) "ResourceProviderAllocationRetrievalFailed ERROR log message on fresh n-cpu startup" [Low,Triaged]
21:10:41 mriedem https://bugs.launchpad.net/nova/+bug/1789998
21:10:50 mriedem happens on every start of a new compute
21:14:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't log an error if attachment_create fails https://review.openstack.org/609547
21:23:03 sorrison mriedem: Missing some context RE: "maybe that's just never been a problem b/c of quota restrictions, the multi-create thing i mean"
21:23:30 mriedem meaning maybe no one has ever felt the need to restrict certain groups of users from being able to make multi-create requests
21:24:00 mriedem because multi-create can be abused, e.g. https://review.openstack.org/#/c/607735/
21:24:41 mriedem sorrison: maybe a better question is, what is the highest any of your tenants have for instance quota?
21:26:46 sorrison mriedem: 2048 is the highest just looking in our DB
21:26:52 mriedem jesus
21:27:03 mriedem have you ever tried to create 2048 servers in a single create request?
21:27:14 sorrison haha don't be silly :-)
21:27:17 mriedem because the API will let you do that
21:27:34 mriedem there is no rate limiting on multi-create requests
21:27:46 mriedem and that size of request will melt your scheduler
21:28:03 mriedem related: https://review.openstack.org/#/c/510235/
21:28:08 sorrison is the multi create number set in the request spec? I can have a look in the db to see what our stats are like
21:28:38 mriedem yes, it's the request spec "num_instances" field
21:35:39 openstackgerrit Merged openstack/os-vif master: clean up ip_command interface https://review.openstack.org/609414
21:36:32 sorrison mriedem: very tricky to get that info out of mysql due to json blob. We're not running a version of mysql that has json support sadly
21:36:51 mriedem damn
21:40:38 melwitt I never knew request spec was a json blob until now O.o
21:40:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't log error in _remove_deleted_instances_allocations if compute is new https://review.openstack.org/609552
21:41:18 sorrison either way, we're aware of the issue but haven't had any major issues
21:41:42 mriedem maybe i can get some data from some public cloud ops
21:41:46 sorrison Just trying to get my sql foo on to see if I can extract the num_instances
21:51:46 sorrison ok so max num_instances we've had is 49
21:52:48 sorrison mriedem: stats here http://paste.openstack.org/show/731867/
21:53:23 melwitt neat
21:56:53 mriedem sorrison: nice, thanks
22:01:13 sorrison mriedem: I sorted out https://review.openstack.org/#/c/608474/ still not sure about correct name for policy as it does affect list and show for a flavor
22:01:55 mriedem that's not checked on show is it?
22:02:05 mriedem or you mean, allow non-admins to show private flavors that they don't have access to?
22:02:21 mriedem i.e. support person trying to triage a bug for a server created with a private flavor?
22:07:40 sorrison Yes
22:07:55 sorrison The change to allow policy for a flavor show is https://review.openstack.org/#/c/608474/3/nova/objects/flavor.py
22:08:10 sorrison it's for reporting scripts
22:09:16 mriedem yup took me a second to sort that out
22:09:31 sorrison yeah it goes down a few layers from the api
22:12:29 mriedem sorrison: ok comments inline;
22:12:41 mriedem i left some suggestions about the rule name, but they aren't awesome
22:12:49 mriedem maybe alex_xu or gmann or dansmith have ideas
22:15:06 sorrison yeah I can't think of something good there
22:16:04 sorrison with disabled flavors I think they are useful. We are planning on retiring some flavors soon and were planning on disabling them by updating the DB
22:16:27 sorrison unless there is a better way to retire flavors?
22:27:33 mriedem there is no way to disable flavors via the API, which is why i'm sort of hesitant to mention them
22:27:49 mriedem but it's not a big deal to leave that in if you're hacking flavors.disabled via the db directly
22:27:55 mriedem it just sucks you have to do that...
22:28:09 mriedem we have a PUT /flavors/{flavor_id} now...seems that would be a place to disable/enable flavors
22:28:16 melwitt AFAIK, people retired flavors by deleting them. and I remember we had a bug back from eons ago where 'nova show' would fail on existing instances with the retired flavor because it was trying to pull a flavor that was deleted from the db
22:28:43 melwitt that got fixed by flavors embedded on instances
22:28:57 mriedem and we expose the flavor details embedded in the instance in the API now
22:34:15 sorrison hmm yeah that makes sense and can prob just retire by deleting now
22:57:36 mriedem efried: see the thread on the ML about moving taskflow out of openstack governance?
22:57:40 mriedem powervm might care about that
22:58:54 mriedem oh heh i see you did :)
22:59:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip _remove_deleted_instances_allocations if compute is new https://review.openstack.org/609552
23:00:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip _remove_deleted_instances_allocations if compute is new https://review.openstack.org/609552
23:44:04 openstackgerrit iain MacDonnell proposed openstack/nova master: Handle online_data_migrations exceptions https://review.openstack.org/608091
#openstack-nova - 2018-10-11
00:05:22 mnaser mnaser
00:31:55 openstackgerrit Brin Zhang proposed openstack/python-novaclient stable/pike: Redirect the old release note url https://review.openstack.org/609566
01:52:21 openstackgerrit Merged openstack/nova master: api-ref: Remove a description in servers-actions.inc https://review.openstack.org/608796
01:52:29 openstackgerrit Merged openstack/nova master: Remove useless TODO section https://review.openstack.org/608802
03:03:39 openstackgerrit Merged openstack/nova master: remove commented-out code https://review.openstack.org/605635
03:06:41 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: Handle volume API failure in post_live_migration https://review.openstack.org/609517
03:19:13 openstackgerrit Tao Li proposed openstack/nova master: Cleanup the instance when MessageDeliveryFailure exception https://review.openstack.org/608500
03:24:01 openstackgerrit Merged openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260
05:53:56 openstackgerrit Sam Morrison proposed openstack/nova master: Allow ability for non admin users to list all flavors. https://review.openstack.org/608474
05:55:26 openstackgerrit Juan Antonio Osorio Robles proposed openstack/nova master: Remove saved_file_rules check from policy https://review.openstack.org/609591

Earlier   Later