Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-16
15:41:10 fried_rice It just makes sense too though.
15:41:25 fried_rice superdan: Are you saying mordred tends to the garrulous?
15:42:00 superdan oh I dunno about that
15:42:15 superdan he's just mordredulous
15:42:26 fried_rice mordredoquacious
15:44:27 mriedem huh, this is kind of fun
15:44:28 mriedem stack@queens:~$ openstack server list
15:44:28 mriedem +--------------------------------------+---------+--------+----------+-------+--------+
15:44:29 mriedem | ID | Name | Status | Networks | Image | Flavor |
15:44:29 mriedem | fd20384d-c0e5-40ab-b9f5-c3fae406379b | server1 | ERROR | | | |
15:44:29 mriedem +--------------------------------------+---------+--------+----------+-------+--------+
15:44:35 mriedem this server failed in the api i think
15:44:38 mriedem on create
15:44:53 mriedem it's volume-backed so that's why no image, but no flavor?
15:45:50 mriedem melwitt: mnaser: heh, as a user, i just hit the bug we're trying to fix,
15:46:09 mriedem volume-backed server create failed, i delete it, then went to do it again but the volume is reserved
15:46:15 mriedem so now i have to switch to admin to force detach it
15:46:40 melwitt that bug so hot right now
15:49:28 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: Allow to configure amount of PCIe ports https://review.openstack.org/545034
15:52:01 cdent I think a t-shirt with "that bug so hot right now" would work pretty well. It rings.
15:54:41 mordred superdan, mriedem, fried_rice: yes please to not having auth on discovery urls
15:55:33 fried_rice cdent: Picture of a cockroach with a smug grin and a pompadour
15:56:02 ingy mordred: hey o/
15:56:16 cdent fried_rice: that will do nicely
16:00:01 finucannot FYI, I'm gone for the next week so if anyone pings me don't expect me to answer. See everyone at the PTG!
16:00:11 ingy cdent is OnIt™
16:00:37 cdent ingy: always
16:01:03 openstackgerrit Stephen Finucane proposed openstack/nova master: Rename '_numa_get_constraints_XXX' functions https://review.openstack.org/385072
16:01:03 openstackgerrit Stephen Finucane proposed openstack/nova master: Standardize '_get_XXX_constraints' functions https://review.openstack.org/385071
16:05:16 mriedem am i making some obvious mistake here?
16:05:17 mriedem curl -d '{"os-force_detach": {}}' -H "accept: application/json" -H "x-auth-token: $token" http://199.204.45.19/volume/v3/e9d773beeef2435eb59f7c6eeaf685a9/volumes/126c8d4b-c582-484a-8c09-fe901a7dc17f/action
16:05:17 mriedem {"badRequest": {"message": "There is no such action: None", "code": 400}}
16:05:21 mriedem https://developer.openstack.org/api-ref/block-storage/v3/#force-detach-a-volume
16:06:38 melwitt did you include a request body?
16:06:58 mriedem yeah, -d
16:07:30 melwitt oh, I'm blind
16:09:33 mriedem aha
16:09:41 mriedem Feb 16 16:08:49 queens devstack@c-api.service[1549]: DEBUG cinder.api.openstack.wsgi [None req-c7279a60-f7ba-4a11-98f2-8fa2b2ec281d demo demo] Unrecognized Content-Type provided in request {{(pid=1723) get_body /opt/stack/cinder/cinder/api/openstack/wsgi.py:724}}
16:09:54 mriedem excellent UX
16:10:18 cdent a != b
16:13:54 mriedem yeah my fault
16:14:54 mriedem cinder api can figure out if i'm missing the content-type header though and let me know
16:15:02 mriedem rather than just 'no such action, f u'
16:15:25 smcginnis if user == mriedem: return "f u"
16:15:37 mriedem why i aughta
16:15:59 melwitt lol
16:23:09 mrjk_ Hi, I hit this problem: https://bugs.launchpad.net/nova/+bug/1579213. Comments are pretty well explicit as well. So I came to change filter order, to lower scheduler_driver_task_period=30 (#was 60) but nothing worked.
16:23:09 openstack Launchpad bug 1579213 in OpenStack Compute (nova) "ComputeFilter fails because compute node has not been heard from in a while" [Undecided,Invalid]
16:23:38 mriedem mrjk_: are you using the caching scheduler?
16:24:07 mrjk_ Now, I only have the service_down_time>60 option, but I don't like it as it will impact all of my services
16:24:12 mrjk_ mriedem, lemme check
16:24:25 mriedem if you're not using the caching_scheduler, scheduler_driver_task_period is not used
16:25:52 mrjk_ mriedem, no caching_scheduler in place (I'm running liberty)
16:26:20 mriedem then scheduler_driver_task_period isn't used
16:26:36 mriedem you're sure that you don't have scheduler_driver set in nova.conf?
16:27:13 mrjk_ Got this: scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
16:28:24 mriedem then the compute filter is probably failing because you have a down compute service
16:28:31 mriedem nova service-list will show you the compute service that is down
16:28:45 openstackgerrit Merged openstack/nova master: trivial: Move __init__ function https://review.openstack.org/538223
16:29:02 mrjk_ Actually it fails when I try to load a lots of VM (from 30)
16:29:44 mrjk_ I have some node down, because it always have, it shouldn't impact. The sheduling process mays definitely take up to 1 minute as well
16:29:59 mriedem mrjk_: ok, well, that could be lots of things potentially so you're going to have to dig through some logs; and you're on a long EOL release
16:30:28 mrjk_ yep, I know ... The log said the host heartbeat is too old
16:31:33 mrjk_ The thing I don't get is how this does work. Does it do like a snapshot of it's current available hosts, and then process it for all instances ?
16:31:58 cfriesen mrjk_: basically, yes
16:33:23 mrjk_ So I've no choice to increase service_down_time :/
16:34:29 cfriesen mrjk_: is the service actually down, or does it just look down due to missed updates from the compute node due to load?
16:36:31 mrjk_ Services are up, I only have few nodes down (3/150). In the log I see the scheduler querying for each compute status, and then after 1 minute it start to say all computes seems dead because no heartbeat
16:37:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a nova-caching-scheduler job to the experimental queue https://review.openstack.org/539260
16:37:38 mrjk_ And I look into the code, the is_up check is made at the query time, it does not seems to query any cache service (my 2cts)
16:38:20 mriedem are you running conductor and scheduler on the same host? do you only have 1 conductor? maybe you need more conductor workers.
16:38:23 mriedem sounds like a scaling problem
16:39:16 mriedem anyway, debugging liberty deployment scaling issues isn't really the focus for this channel, you can try #openstack or #openstack-operators maybe
16:39:25 cfriesen mriedem: if you were scheduling a whole bunch of instances, such that by the time you get to the last one the cached "last checkin" time on a service was more than "service_down_time" ago, wouldn't that cause this sort of thing?
16:40:02 mriedem possible, idk, i don't create a bunch of instances in a single request
16:40:09 mriedem i know doing so has all sorts of issues
16:40:30 cfriesen mrjk_: is there a reason why you are creating so many in one request?
16:40:42 mriedem like if you create 1000 instances in a single request, we don't limit that, and we can cause the rpc call from conductor to scheduler to timeout and retry it, thus increasing the load and failure
16:41:06 mriedem https://review.openstack.org/#/c/510235/
16:44:02 cfriesen mrjk_: as a general rule, I'd suggest keeping the number of servers in a single boot request small enough that the scheduling time is safely below "service_down_time"
16:48:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide a hint when performing a volume action can't find the method https://review.openstack.org/545382
16:56:52 mriedem melwitt: i asked laura last night if she knew what "get on the horn" meant and she had no idea what i was talking about
16:58:45 superdan wat
16:59:04 melwitt hah, *solidarity*
17:04:03 giblet just a quick heads up, I will be mostly unavailable during next week. see you in Dublin!
17:04:09 cfriesen melwitt: given the new behaviour in https://review.openstack.org/#/c/528385/ it seems the new expectation if you fail to build is that you need to create a new cinder volume and create a new instance. previously you could try doing a "rebuild" operation after fixing whatever the problem was.
17:06:08 mriedem cfriesen: what is the new behavior here? that the volume isn't left stuck in a non-available status?
17:06:54 melwitt cfriesen: it would only be a new volume if it was set to delete_on_termination. otherwise, it's just detached. and yes, create new instance
17:07:17 cfriesen mriedem: about the instance.info_cache.network_info vs self.network_api.get_instance_nw_info(context, instance) you said the method on the instance is a helper method...but instance.info_cache isn't a method, it's an object.
17:07:35 melwitt tbh, I wasn't thinking of anyone using rebuild to fix a failed build
17:08:01 mriedem melwitt: not necessarily a new volume
17:08:11 cfriesen mriedem: I think previously you would have an instance in ERROR state with an attached volume. I think you could have done a "rebuild" on it from the error state.
17:08:15 mriedem melwitt: i can create a volume in cinder, and use it for bfv, and specify delete_on_termination
17:08:19 mriedem i'm not sure why you'd do that though
17:08:51 mriedem cfriesen: i thought you were asking about instance.get_network_info()
17:08:58 mriedem not self.network_api.get_instance_nw_info(context, instance)
17:09:03 mriedem self.network_api.get_instance_nw_info(context, instance) will rebuild the info cache
17:09:07 melwitt mriedem: if delete_on_termination was set and it failed the build and you had to delete the instance and start again, you would have to create a new volume, right?
17:09:14 mriedem instance.get_network_info() is the same as instance.info_cache.network_info
17:09:26 mriedem melwitt: yes

Earlier   Later