Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-13
16:19:50 artom melwitt, will take a look tomorrow
16:20:03 melwitt thanks!
16:20:20 artom Hopefully before then, actually
16:29:51 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Record the machine_type of instances in system_metadata https://review.opendev.org/c/openstack/nova/+/767533
16:29:55 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-manage: Add commands for managing instance machine type https://review.opendev.org/c/openstack/nova/+/769548
16:29:57 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-status: Add hw_machine_type check for libvirt instances https://review.opendev.org/c/openstack/nova/+/770643
16:56:44 sean-k-mooney artom: i havenet made any changes yet but i responed to your questions in https://review.opendev.org/c/openstack/nova-specs/+/764999/2/specs/wallaby/approved/libvirt-vdpa-support.rst can you re review
16:57:09 sean-k-mooney i have some WIP patches up as well https://review.opendev.org/q/topic:%22vhost-vdpa%22+(status:open%20OR%20status:merged)
16:57:27 openstackgerrit Balazs Gibizer proposed openstack/nova master: DNM try to replace retry with libvirt event in detach https://review.opendev.org/c/openstack/nova/+/770246
16:57:45 sean-k-mooney its not fully complete as i still need to extend the pci tracker and where we create the pci request
16:58:01 sean-k-mooney and i need to test it locally and write tests and docs
16:58:28 sean-k-mooney but it has the outline of about 2 thirds of the code
16:59:26 artom sean-k-mooney, cool - I want to un-WIP my socket affinity spec, then I'll take a look
17:10:04 markguz_ Hi nova folks. THe good people at #openstack-ironic thought it might be a good idea for me to ask about my problem here.
17:12:04 markguz_ I've got an issue where my ironic instance spawning is getting stuck for +/- 10mins just after i start deployment. The nova-scheduler picks the correct compute node and then the nova-compute node reports "Starting instance... _do_build_and_run_instance"
17:12:40 markguz_ then nothing happens for about 10 mins. then suddenly the process starts and deployment continues..
17:13:30 markguz_ TheJulia over at ironic thinks that the process is getting stuck at the scheduling stage. The vm reports Building, but the task status sits at "none" during that 10mins
17:14:43 markguz_ For VMs there is no delay, only for BMs. This is Rocky. and it is not a busy deployment. Very little activity going. We can go days without spawning bm or vm instances
17:15:35 markguz_ I've been trying to dig around the code to see what happens when the compute node reports "Starting instance... _do_build_and_run_instance" but it's very hard to follow
17:16:43 markguz_ If anyone could help me follow the rabbit through the rabbit hole and see exactly what is happening, I'd be most appreciative
17:23:58 melwitt markguz_: if it's landed on the compute node already, we don't consider that to be "scheduling" as it's already been scheduled/placed. but to dig into this further you'll want to trace the request id of the line that says "Starting instance" in the nova-compute log and see if you can see where it stops making progress. that would help
17:25:31 sean-k-mooney if this is ironic by they way whe n we get do do_build_and_run_instance at some point the compute manager will hand of to the ironic driver which will call ironic to provision the node
17:26:27 melwitt right. would want to look and see if he can verify it's gotten to that point
17:26:58 sean-k-mooney https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L2186 starting to build instance is right at the top
17:27:31 sean-k-mooney the we save the task state at None and vmstate building
17:27:41 melwitt yeah, I know. just saying he can trace the request id to see how far it gets
17:27:54 melwitt after that
17:29:09 markguz_ here's a grep of the req id for an instance out of the logs http://paste.openstack.org/show/801601/
17:29:44 markguz_ nothing between 9.40 and 10.05
17:31:14 sean-k-mooney do you have the concurnet build limit set
17:31:15 stephenfin melwitt: Comments left
17:31:17 stephenfin sean-k-mooney: ^
17:31:20 sean-k-mooney for the compute service
17:31:26 sean-k-mooney it defaults to 10 i belive
17:31:40 TheJulia markguz_: something between scheduling and initial network setup :\
17:31:55 sean-k-mooney ock "compute_resources" acquired by "nova.compute.resource_tracker.instance_claim" :: waited 1495.269s
17:32:14 sean-k-mooney it looks like it was jsut waiting on the RT lock
17:32:20 melwitt yep, waited 24 min for the lock
17:32:47 melwitt thank you stephenfin
17:33:29 markguz_ why would the lock take 25mins?
17:34:15 sean-k-mooney the comput service is likely bussy starting up
17:34:24 sean-k-mooney you mentioned this is only after the iniall start right
17:34:41 sean-k-mooney or is this for each spawn
17:34:59 lyarwood just grep for the compute_resources lock and see what was holding it before?
17:35:35 sean-k-mooney ya that too
17:35:45 sean-k-mooney you could see how many other instance got the lock in that interval
17:36:22 markguz_ sean-k-mooney: it's every spawn. usually 10mins, sometimes longer and sometimes shorter
17:36:55 lyarwood does the resource tracker make external API calls in the Ironic driver?
17:38:46 sean-k-mooney the RT is shared so i dont think so
17:39:47 lyarwood yeah sorry I mean the code that refreshes it within the driver
17:40:11 sean-k-mooney im pretty sure this is the lock in question https://opendev.org/openstack/nova/src/branch/stable/rocky/nova/compute/manager.py#L2221-L2222
17:41:13 melwitt markguz_: I think you might be hitting https://bugs.launchpad.net/nova/+bug/1864122
17:41:15 openstack Launchpad bug 1864122 in OpenStack Compute (nova) "Instances (bare metal) queue for 30-60 seconds when managing a large amount of Ironic nodes" [Medium,Fix released] - Assigned to Jason Anderson (jasonandersonatuchicago)
17:41:50 sean-k-mooney yep that was what grabed the lock https://opendev.org/openstack/nova/src/branch/stable/rocky/nova/compute/resource_tracker.py#L159-L160
17:42:13 markguz_ i have +/- 240 nodes
17:42:24 markguz_ is that a large amount?
17:42:37 sean-k-mooney melwitt: yep a race on the lock with the update periodic task seams likely
17:42:41 melwitt if you read the bug it says can be seen around > 100 nodes
17:43:12 sean-k-mooney markguz_: how many ironic compute services do you have
17:43:15 melwitt that fix is available in ussuri and onward, it was not backported because it requires a newer version of oslo.concurrency
17:43:24 markguz_ sean-k-mooney: 1
17:43:29 sean-k-mooney i belive the periodic will only update the resouce usage for the nodes that are assgined to it
17:44:10 sean-k-mooney so i think you can scale it by deploying more compute service instances TheJulia is that correct?
17:45:03 sean-k-mooney markguz_: if you have 3 contolers i would suggest running an ironic nova compute service instance on each assuming that makes sense to TheJulia or others
17:45:07 TheJulia sean-k-mooney: yes, you can, you should just be able to run multiple instances
17:45:19 TheJulia markguz_: ^^^ instances of nova-compute configured for ironic
17:45:33 stephenfin melwitt: comments left on the bug report too
17:46:09 sean-k-mooney markguz_: the other thing you could do is reduce the interval of the periodic
17:46:21 melwitt hm, I thought you needed to configure node partitioning to do that
17:46:23 sean-k-mooney we fixed it by chanigin the type of lock we use
17:46:46 melwitt "conductor groups"
17:46:52 TheJulia melwitt: only to force specific grouping/allocation into specific grouping
17:46:56 sean-k-mooney that on the ironic side i think
17:47:01 melwitt it's not
17:47:05 TheJulia its on both sides
17:47:12 sean-k-mooney ah ok
17:47:19 melwitt well, it might be but you have to do it on the nova side too
17:47:19 markguz_ peridoc_task_interval is set to 240
17:47:24 TheJulia otherwise it runs a hash ring based upon the node list
17:47:31 TheJulia and the group is just a key in the hash ring
17:48:01 sean-k-mooney we improved this in nova by using oslos fair locks
17:48:02 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/711528/2
17:48:02 TheJulia the nova side name is a little different because naming_is_fun^TM
17:48:23 markguz_ so we use this in a lab env and when we spin up baremetal we need to spin up a spcific node as they are connected to specific hardware that is being tested
17:48:39 sean-k-mooney but that was only done in ussuri
17:48:48 TheJulia sean-k-mooney: ohhhhh neat
17:49:02 sean-k-mooney so we would have to backport it unfortunetlly im not sure oslo has the required support in rocky let me check
17:49:13 melwitt ok conductor groups are not available until stein anyways
17:49:30 sean-k-mooney we would need oslo.concurrancy 3.29.0 to backport it
17:49:39 markguz_ if i run multiple computes i'm guessing that i will need to change how i call a instance. right now i use the "avail_zone:compute_host:bm_uuid" trick
17:49:40 melwitt yeah, I said all of that earlier
17:49:55 sean-k-mooney stable rocky is oslo.concurrency===3.27.0
17:50:07 TheJulia markguz_: yeah, :\
17:50:14 melwitt yes, the patch that added fair locks bumped the oslo.concurrency version
17:50:22 sean-k-mooney so it can go back to stien
17:50:32 sean-k-mooney but not rocky
17:50:40 melwitt so it wasn't bumped until ussuri
17:51:18 sean-k-mooney markguz_: ya you would need to know which host has it
17:51:33 markguz_ what's ironic is (pun intended) is that i was upgrading with the intention of getting to ussuri but when ironic failed at the rocky step i didn't want to compound the problem by continuing to upgrade
17:52:37 melwitt huh yeah actually it could be backported to stein because the upper constraint is 3.29.1 for whatever reason

Earlier   Later