Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-16
16:26:20 mriedem then scheduler_driver_task_period isn't used
16:26:36 mriedem you're sure that you don't have scheduler_driver set in nova.conf?
16:27:13 mrjk_ Got this: scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
16:28:24 mriedem then the compute filter is probably failing because you have a down compute service
16:28:31 mriedem nova service-list will show you the compute service that is down
16:28:45 openstackgerrit Merged openstack/nova master: trivial: Move __init__ function https://review.openstack.org/538223
16:29:02 mrjk_ Actually it fails when I try to load a lots of VM (from 30)
16:29:44 mrjk_ I have some node down, because it always have, it shouldn't impact. The sheduling process mays definitely take up to 1 minute as well
16:29:59 mriedem mrjk_: ok, well, that could be lots of things potentially so you're going to have to dig through some logs; and you're on a long EOL release
16:30:28 mrjk_ yep, I know ... The log said the host heartbeat is too old
16:31:33 mrjk_ The thing I don't get is how this does work. Does it do like a snapshot of it's current available hosts, and then process it for all instances ?
16:31:58 cfriesen mrjk_: basically, yes
16:33:23 mrjk_ So I've no choice to increase service_down_time :/
16:34:29 cfriesen mrjk_: is the service actually down, or does it just look down due to missed updates from the compute node due to load?
16:36:31 mrjk_ Services are up, I only have few nodes down (3/150). In the log I see the scheduler querying for each compute status, and then after 1 minute it start to say all computes seems dead because no heartbeat
16:37:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Add a nova-caching-scheduler job to the experimental queue https://review.openstack.org/539260
16:37:38 mrjk_ And I look into the code, the is_up check is made at the query time, it does not seems to query any cache service (my 2cts)
16:38:20 mriedem are you running conductor and scheduler on the same host? do you only have 1 conductor? maybe you need more conductor workers.
16:38:23 mriedem sounds like a scaling problem
16:39:16 mriedem anyway, debugging liberty deployment scaling issues isn't really the focus for this channel, you can try #openstack or #openstack-operators maybe
16:39:25 cfriesen mriedem: if you were scheduling a whole bunch of instances, such that by the time you get to the last one the cached "last checkin" time on a service was more than "service_down_time" ago, wouldn't that cause this sort of thing?
16:40:02 mriedem possible, idk, i don't create a bunch of instances in a single request
16:40:09 mriedem i know doing so has all sorts of issues
16:40:30 cfriesen mrjk_: is there a reason why you are creating so many in one request?
16:40:42 mriedem like if you create 1000 instances in a single request, we don't limit that, and we can cause the rpc call from conductor to scheduler to timeout and retry it, thus increasing the load and failure
16:41:06 mriedem https://review.openstack.org/#/c/510235/
16:44:02 cfriesen mrjk_: as a general rule, I'd suggest keeping the number of servers in a single boot request small enough that the scheduling time is safely below "service_down_time"
16:48:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide a hint when performing a volume action can't find the method https://review.openstack.org/545382
16:56:52 mriedem melwitt: i asked laura last night if she knew what "get on the horn" meant and she had no idea what i was talking about
16:58:45 superdan wat
16:59:04 melwitt hah, *solidarity*
17:04:03 giblet just a quick heads up, I will be mostly unavailable during next week. see you in Dublin!
17:04:09 cfriesen melwitt: given the new behaviour in https://review.openstack.org/#/c/528385/ it seems the new expectation if you fail to build is that you need to create a new cinder volume and create a new instance. previously you could try doing a "rebuild" operation after fixing whatever the problem was.
17:06:08 mriedem cfriesen: what is the new behavior here? that the volume isn't left stuck in a non-available status?
17:06:54 melwitt cfriesen: it would only be a new volume if it was set to delete_on_termination. otherwise, it's just detached. and yes, create new instance
17:07:17 cfriesen mriedem: about the instance.info_cache.network_info vs self.network_api.get_instance_nw_info(context, instance) you said the method on the instance is a helper method...but instance.info_cache isn't a method, it's an object.
17:07:35 melwitt tbh, I wasn't thinking of anyone using rebuild to fix a failed build
17:08:01 mriedem melwitt: not necessarily a new volume
17:08:11 cfriesen mriedem: I think previously you would have an instance in ERROR state with an attached volume. I think you could have done a "rebuild" on it from the error state.
17:08:15 mriedem melwitt: i can create a volume in cinder, and use it for bfv, and specify delete_on_termination
17:08:19 mriedem i'm not sure why you'd do that though
17:08:51 mriedem cfriesen: i thought you were asking about instance.get_network_info()
17:08:58 mriedem not self.network_api.get_instance_nw_info(context, instance)
17:09:03 mriedem self.network_api.get_instance_nw_info(context, instance) will rebuild the info cache
17:09:07 melwitt mriedem: if delete_on_termination was set and it failed the build and you had to delete the instance and start again, you would have to create a new volume, right?
17:09:14 mriedem instance.get_network_info() is the same as instance.info_cache.network_info
17:09:26 mriedem melwitt: yes
17:10:17 melwitt that's what I was saying to cfriesen, you would only have to create a new volume if you had used delete_on_termination. else the volume would be only detached and you could use it again with a new instance
17:10:38 smcginnis You might want a bfv deleted on instance delete if you are using it just to manage your storage capacity separate from your local n-compute local storage.
17:11:07 cfriesen mriedem: how do I know when I can call instance.get_network_info()? Specifically I'm looking at nova.compute.rpcapi.pre_live_migration()....I want to adjust the RPC timeout based on the number of network ports.
17:11:12 mriedem smcginnis: so not the root disk
17:11:15 mriedem application data
17:11:30 mriedem i just figure people that create a volume directly in cinder and use it to bfv care about re-using the volume,
17:11:36 mriedem and people that let nova create the volume for you, don't care
17:11:47 smcginnis Eh, less useful maybe just for application data, but still can be used in that way.
17:12:00 mriedem cfriesen: you can call it at any time...?
17:12:13 smcginnis mriedem: I would think that is usually the case that they do care about that data though if they create in cinder first.
17:12:23 mriedem smcginnis: yeah
17:12:57 mriedem cfriesen: presumably we rebuild the nw info cache before starting live migration anyway
17:13:21 mriedem cfriesen: yes we do
17:13:22 mriedem network_info = self.network_api.get_instance_nw_info(context, instance)
17:13:28 mriedem in pre_live_migration in the ComputeManager
17:13:30 cfriesen mriedem: I'm confused (clearly). in the existing code, in some places where they want network_info they look at instance.info_cache.network_info, and in other places they call
17:13:32 mriedem that will refresh the nw info cache
17:13:32 cfriesen self.network_api.get_instance_nw_info(
17:13:33 cfriesen context, instance)
17:13:38 cfriesen whoops, bad paste, sorry
17:13:47 mriedem so at that point, the instance.info_cache.network_info is up to date
17:13:55 mriedem so if you're going to use it to count ports to adjust rpc call timeout, you should be good
17:15:13 cfriesen no, I need to have the ports in the rpc code that is *calling* pre_live_migration on the dest node.
17:15:23 cfriesen so updating in pre_live_migration doesn't help
17:15:36 mriedem let me dig up the live migration call chart
17:15:45 mriedem of which i still owe fried_rice a shiny nickel
17:16:01 fried_rice \o/
17:16:02 cfriesen I think live_migration() on the source node calls pre_live_migration() on the dest
17:16:39 cfriesen or rather _do_live_migration() I guess
17:17:16 mriedem f me if i can ever find anything in our docs
17:17:26 mriedem https://docs.openstack.org/nova/latest/reference/live-migration.html
17:17:59 mriedem cfriesen: yeah you're right
17:18:07 openstackgerrit Matthew Edmonds proposed openstack/nova-specs master: PowerVM Virt Integration (Rocky) https://review.openstack.org/545111
17:18:34 mriedem and _do_live_migration doesn't refresh the instance nw info cache before calling pre_live_migration
17:18:50 cfriesen okay, so I probably need to call self.network_api.get_instance_nw_info(context, instance) ?
17:18:57 mriedem cfriesen: so from _do_live_migration if you call instance.get_network_info() then you're just getting whatever is current for the instance in the db
17:19:06 mriedem if you really want to refresh to get the latest
17:19:27 cfriesen what would cause that to be stale? if someone was in the middle of attaching an interface or something?
17:19:31 mriedem but we have the network events from neutron if that changes, and the heal_instance_info_cache periodic
17:19:44 mriedem umm
17:20:10 mriedem yeah,
17:20:23 mriedem since we don't set a task_state for attaching/detaching an interface, you can live migrate the server at the same time
17:20:36 mriedem if the instance.task_state was not None (attaching/detaching), you couldn't do a live migration
17:20:58 mriedem thta's something i've always wondered about,
17:21:09 mriedem why we don't change task_state while attaching/detaching interfaces and volumes
17:22:30 cfriesen might be a fun stress test. ping-pong live migrations while attaching/detaching interfaces/volumes
17:24:46 mriedem fun like a hernia
17:26:51 mriedem cfriesen: so you're probably better off just starting small and getting the current info cache in _do_live_migration and basing the rpc timeout on that
17:36:11 cfriesen mriedem: agreed, it should be good enough. Is this idea of varying the RPC timeout based on number of attached interfaces something that you think would be generally useful? We're seeing it take ~1.5 sec per interface in pre_live_migration() to update the ports, but I'm not sure if that's typical or due to our neutron changes.
17:36:57 superdan I don't really like the idea of doing that
17:37:16 cfriesen and in nova those calls to neutron are serialized, so with a large number of interfaces it's possible to eat up a good chunk of the RPC timeout just doing the port updates
17:37:17 superdan I'd rather see something in olso.messaging that heartbeats running calls so we have a soft and hard timeout range
17:37:17 mriedem i seem to remember having similar conversations for other rpc calls that can take a long time, but can't remember details,

Earlier   Later