Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-17
12:02:44 sean-k-mooney some of the operation in https://github.com/openstack/os-brick/blob/5902166149ba46da5f36eb352e62f3361269fb2c/os_brick/initiator/connectors/lightos.py#L107 also do io which will cause eventlet to yeild but in genrealy i dont think we want to allow that to run in a thread in the compute agent
12:04:24 sean-k-mooney lightos is not the only problematic driver that is now tryign to do this
12:05:30 sean-k-mooney but i really dong like this design parrent in os-brick and i think all montors/deamon threads shoudl be seperate processes or not part of os-brick at all
12:07:13 sean-k-mooney ... os-brick is doing this in several places.... that is really not good we should probaly make this a ptg topic
12:19:32 yuval ok, thanks for the feedback, we will think what our options
12:23:41 sean-k-mooney gibi: by the way nova still has root wrap solely for os-brick right. is there a reason we cant move this to os-brick?
12:23:43 sean-k-mooney https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters
12:25:36 sean-k-mooney gibi: looking at what os-brick is doing really make me think we shoudl not be loading it in the compute agent directly since its problematic in several ways
12:26:11 gibi I think only os-brick using rootwrap we converted the rest of privsep
12:26:16 gibi so I think we can move it
12:26:27 gibi but probably we need a volunteer
12:26:57 gibi as of what os-brick does under the hood, I'm not that familiar with. I saw that iscsi connector also spawn threads so I assume that works other
12:26:57 sean-k-mooney yep its just techdebth
12:27:12 gibi otherwise we would have failures
12:27:23 sean-k-mooney it works but they are green threads
12:27:27 sean-k-mooney not native threads
12:27:36 sean-k-mooney because we monkey patch and its running in the same process
12:27:50 sean-k-mooney os-brick does not use eventlets iteself
12:28:08 sean-k-mooney which is true for os-vif alos both both inherit it becasue they are imported into nova which does
12:28:46 gibi ahh you have a good point
12:28:58 gibi those just look like native threads but we monkey patched them
12:29:05 gibi but then I don't fully get the problem
12:29:21 gibi why don't they spawn greenlets freely?
12:29:25 gibi under the hood
12:29:52 sean-k-mooney gibi: simple since they are monkey patched if they dont yeild nova's event loop is blocked
12:30:11 gibi they have thight loops without io?
12:30:22 sean-k-mooney yes in some places
12:30:31 gibi ahh so they have some thight loops
12:30:41 gibi then those needs some careful yields
12:30:59 sean-k-mooney https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/lightos.py#L164-L182
12:31:38 sean-k-mooney so this yeild because monitor_db does some writes to a file and those sleeps
12:32:11 sean-k-mooney monitor_message_queue wont yeild unless the log satement casue it to happen
12:32:29 sean-k-mooney but it will terminate when the queue is empty
12:32:52 gibi I see
12:33:41 sean-k-mooney isci is potentally worse
12:33:43 sean-k-mooney https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L723-L780
12:34:04 sean-k-mooney it spawns a bunch of threds then joins in the multipath code
12:34:47 sean-k-mooney the join shoudl cause a yeild and those threads will be interleaved with nova
12:35:11 sean-k-mooney but this type of raw threadign is really not a good thing with eventlets in the mix
12:35:31 gibi jeah that seem a lot more complex. but still it works or we would have strange bugs
12:35:36 sean-k-mooney they also have sleeps https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L777
12:36:35 sean-k-mooney ya so that will do io an dthe sleep will yeild which the do on each iteration
12:37:11 sean-k-mooney i just think the use of thread in general in os-brick is problematic
12:45:37 opendevreview Imran Hussain proposed openstack/nova master: [nova/libvirt] Support for checking and enabling SMM when needed https://review.opendev.org/c/openstack/nova/+/825496
14:15:22 IPO Hello, all ! What are the next steps toward https://review.opendev.org/c/openstack/nova/+/805649 ? Or it is ok,it is needed just wait a bit ?
14:18:37 gibi IPO: I guess we need a second core
14:18:58 gibi sean-k-mooney: you were +2 before on that patch ^^
14:26:57 IPO gibi: I see, thanks for info !
14:42:29 yuval_ sean-k-mooney other the monitor thread - anything is a concern?
15:05:15 sean-k-mooney yuval_: that is my main one.
15:06:32 sean-k-mooney yuval_: gibi listed other issue liek py2 support ectra inline
15:07:15 sean-k-mooney yuval_: so other then the monitor thread and the comments that gibi has already reaied i dont really have other issues with it
15:11:52 bauzas melwitt: gibi: question about InstanceMapping records vs. shelved-offloaded instances and how we quota them
15:12:01 bauzas melwitt: gibi: https://review.opendev.org/c/openstack/nova/+/713301/34/nova/limit/placement.py#80
15:12:42 bauzas iirc, we don't have quota usage for shelved-offloaded instances, right?
15:12:43 sean-k-mooney shelved instance shoudl consume the same quota for disk as running but they should not consume quota for cpu or ram
15:13:12 sean-k-mooney bauzas: it depend on if it BFV really i think
15:13:22 bauzas sean-k-mooney: my question is about the instances/servers quota usage
15:13:38 sean-k-mooney well i t should consume isntance quota
15:14:13 sean-k-mooney that is a limit on the number of non deleted instance you can have in the db
15:14:16 bauzas surely, but I don't remember about the current behaviour
15:14:18 sean-k-mooney its independt fo the state
15:14:34 sean-k-mooney it wont be tracked in placment currently
15:14:42 sean-k-mooney since we dont consume allocations of instance
15:14:47 sean-k-mooney but i guess
15:14:57 sean-k-mooney you coudl count allocation by consumer type maybe
15:15:39 sean-k-mooney when its in shelve offloaded i dont know if we still have any allcoation in placment at all
15:15:51 sean-k-mooney if we had local disk i think we should for storage
15:16:00 sean-k-mooney e.g. if you were usign the rbd image backend
15:16:15 sean-k-mooney but sicne we dont have a host it might not be there
15:16:46 sean-k-mooney melwitt: how does that work with unified limits?
15:17:22 bauzas sean-k-mooney: look at the change
15:17:33 bauzas sean-k-mooney: there is a TODO for consumer types
15:17:44 bauzas sean-k-mooney: for the moment, we look at instance mappings
15:17:48 bauzas hence my question
15:17:51 sean-k-mooney right but consuemr types is not enough
15:18:08 sean-k-mooney a shelve offloaded instance does not have a host
15:18:23 sean-k-mooney so it cant have allcoation against an rp today
15:18:46 sean-k-mooney since the only resouce usage it has are floating ips, and disk really
15:19:32 sean-k-mooney so without creating a RP per compute with inventories of instance,floating ip ectra we cant tack the useage in palcment
15:19:52 sean-k-mooney so we woudl have to tack this in keystone?
15:20:25 bauzas I don't know, we haven't discussed it in https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/unified-limits-nova.html
15:20:40 bauzas tbc, I don't want to trample this series because of this question
15:20:55 sean-k-mooney well for now we can just count it in nova
15:21:05 bauzas my only concern is about making sure we continue to have the same behaviour from the before
15:21:53 sean-k-mooney right so we shoudl continue to count instnace using the instance cell mapings in the nova api db proably for yoga
15:22:38 sean-k-mooney in Zed we could eitehr create an RP per porject with invetoires of diffent quata type like instance or tack it somewhere else
15:23:03 sean-k-mooney the question really is should you be able to get all usage just by looking at placment or not
15:25:29 sean-k-mooney bauzas: ""All the resource class usage will be counted using placement, but server count will make use of instance mappings.""
15:25:35 sean-k-mooney https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/unified-limits-nova.html#enforcing-unified-limits
15:26:17 bauzas sean-k-mooney: nah I was saying about shelved instances
15:26:25 sean-k-mooney yes
15:26:32 sean-k-mooney that fact its shelved does nto change that
15:26:41 bauzas ok, then it was a nit
15:28:27 sean-k-mooney i honestly dont see an impedment to counting server ectra via sharing rps at some point in the futre if we need too or other things like floating ip that are out of scope of the current spec
15:30:25 gibi I agree that offloaded servers if holding floating ips then it should consume quota
15:30:55 gibi and yes we cannot count that with consumer types as no instance allocation exists for offloaded servers in placement
16:01:42 melwitt bauzas, sean-k-mooney: the tl;dr is that unified limits is not meant to have the same behavior as before. shelved offloaded will not consume quota
16:02:08 bauzas melwitt: cool then, I'll switch to +2
16:09:36 melwitt bauzas: I dunno if you've seen this but this is one place where we document the expected differences when counting from placement https://docs.openstack.org/nova/latest/admin/quotas.html#quota-usage-from-placement

Earlier   Later