Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-13
16:38:05 stephenfin there are a few
16:38:34 stephenfin in this scenario that I'm following, you'll be able to use hw:cpu_dedicated_mask *or* hw:cpu_realtime_mask
16:38:59 sean-k-mooney stephenfin: no you would use both
16:39:01 stephenfin I see no reason to say these cores are shared, these are dedicated but non-realtime, and these are dedicated and realtime
16:39:04 sean-k-mooney well optinally
16:39:16 sean-k-mooney e.g. not an exclucive or
16:39:41 stephenfin why? What real-world user is going to use all three types of core in an instance?
16:39:41 sean-k-mooney i dont see a reason to block it
16:39:49 stephenfin Because it's less complicated
16:39:55 sean-k-mooney its more complicated
16:40:20 sean-k-mooney the validation logic to prevent all 3 is extra logic we dont need if we allow it
16:40:45 stephenfin A|B is easier grok than A.issubset(B)
16:40:56 stephenfin and the it makes my XML generation easier
16:41:02 stephenfin s/the //
16:41:04 sean-k-mooney stephenfin: you are over loading hw:cpu_realtime_mask
16:41:18 sean-k-mooney its behavior would change based on the hw:cpu_policy
16:41:29 sean-k-mooney so that gets harder to reason about
16:41:36 stephenfin nope, it stays the same: these are cores that are real-time
16:41:39 sean-k-mooney if we allow both it does not
16:41:41 stephenfin what changes is what happens to the other cores
16:41:53 stephenfin and that's purely based on hw:cpu_policy
16:42:10 sean-k-mooney no it chacnge form tehse are realtime to these are realtime and dedicated and the rest flaot
16:42:17 kashyap sean-k-mooney: Not entirely; there's also a quirk of making sure to specify the interrupt controller (as the default is less featureful) -- `-machine gic-version=max`
16:42:29 kashyap melwitt: For later, added my notes in the change.
16:42:32 stephenfin they're always realtime and dedicated
16:42:39 stephenfin you can't have realtime floating cores
16:42:45 melwitt kashyap: thanks
16:43:05 sean-k-mooney well libvirt allows you to but that is a seperate thign
16:43:14 kashyap Yep
16:43:14 stephenfin ...in nova
16:43:20 sean-k-mooney stephenfin: my perference would be to allow both to be set
16:43:38 stephenfin I could allow it, but I don't want to force it
16:43:46 sean-k-mooney and always require hw:dedicated_cpu_mask for mixed
16:43:58 stephenfin and because I don't want to force it, I'd rather say there's only one way to do this
16:44:14 sean-k-mooney which one do you not want to force
16:44:32 sean-k-mooney the realtime mask or the dedicated mask
16:44:33 stephenfin having to set both hw:cpu_dedicated_mask and hw:cpu_realtime_mask
16:44:46 sean-k-mooney right so i would make the cpu_realtime_mask optional
16:44:50 stephenfin if you want a real-time instance with the non-realtime cores floating
16:44:55 sean-k-mooney and always require the cpu_dedicated_mask
16:45:18 stephenfin but then you have a difference of behavior elsewhere
16:45:28 sean-k-mooney and make it so that if you dont set cpu_realtime_mask but do set hw:cpu_realtime=true then we use the hw:dedciated_cpu_mask
16:45:41 stephenfin now you have to use 'hw:cpu_realtime_mask' when using dedicated
16:45:52 stephenfin but use 'hw:cpu_dedicated_mask' when using mixed
16:46:48 stephenfin anyway, we'll invariably debate this in the patch so back to my original question
16:46:52 sean-k-mooney i think its a much simpler rule to say if you want realtime always use realtime_mask and if you want mixed always use the dedicated_mask
16:47:18 sean-k-mooney if you want both use both and require the realtime mask must be a subset of the dedicated mask
16:47:32 stephenfin if we're doing this, do we want to fix that annoying thing where hw:cpu_realtime_mask has to be preceded by carat?
16:47:43 stephenfin because I don't want to force that 'hw:cpu_dedicated_mask'
16:47:54 stephenfin and it would be nice for them to behave similarly
16:48:05 sean-k-mooney stephenfin: ya i would not mind doing that
16:48:23 sean-k-mooney stephenfin: we said the dedicated mask should follow the rules for the config opntion
16:48:30 sean-k-mooney not the rules for the realtime mask
16:48:33 stephenfin the only reason I see to not do that is jaypipes wanted us to kill 'hw:cpu_realtime_mask' in favor of 'hw:cpu_realtime_set'
16:48:42 stephenfin but tbh, I don't think it's worth the effort
16:48:55 stephenfin we haven't deprecated flavor extra specs before. I don't even want to get into that
16:49:06 sean-k-mooney i would be ok with that i guess but yat that ^
16:49:10 stephenfin ditto for image metadata props, for that matter
16:49:33 stephenfin okay, sweet
16:49:40 stephenfin I can revive those patches so
16:49:43 stephenfin whoo, rebase fun!
16:50:31 sean-k-mooney stephenfin: cool. i prefer each option to do one thing and one thing only. but do what you think is best
16:50:42 sean-k-mooney its a preference not a blocker for me
16:51:10 sean-k-mooney and sice we cant do cross extra spec validation in your validation propsoeal i also prefer that form the avlidation point of view
16:51:36 sean-k-mooney let me know when you want me to review and or play around with it
16:51:47 stephenfin will do
16:51:47 sean-k-mooney im going to drop soon just an fyi
16:51:58 stephenfin ah yeah, it'll be next week anyway
17:16:39 openstackgerrit Elod Illes proposed openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033
17:19:16 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
17:21:15 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036
17:29:09 openstackgerrit Merged openstack/nova stable/rocky: Remove global state from the FakeDriver https://review.opendev.org/712751
17:29:16 melwitt lyarwood: yay, thanks for looking at that. is the same thing happening with the intermittent failures on master?
17:29:56 melwitt *intermittent nova-live-migration failures
17:30:39 lyarwood melwitt: I don't recall seeing this on master but let me have a quick look in logstash
17:31:25 melwitt lyarwood: ok, was just wondering if you knew off the top of your head. I've seen occasional failures of nova-live-migration on master and just curious if it's the same thing. I hope so, cause that would mean your fix would fix that too
17:37:12 lyarwood melwitt: yeah, I can't see anything on master but my elasticsearch foo is awful
17:37:35 melwitt ok np
17:39:38 lyarwood melwitt: FWIW your https://review.opendev.org/#/c/712226/ change failed because of http://status.openstack.org/elastic-recheck/#1813789
17:39:51 lyarwood melwitt: I've not had time to look into that but I have been seeing that across master
17:40:30 melwitt lyarwood: ah thanks. I've been looking at http://status.openstack.org/elastic-recheck/#1844929 again yesterday and today
17:41:06 melwitt so far, no dice
17:42:52 lyarwood melwitt: kk, these all appear rather tricky
17:43:14 melwitt yeah :(
18:28:10 melwitt zzzeek: could you pls sanity check me on this -- this logging is done after getting a response back from the database server right? it's not a client side logging before making the query https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/engine/result.py#L1579
18:28:54 zzzeek melwitt: row logging is after we've executed the statement and we've received rows back from the DBAPI cursor, that line logs the row itself
18:29:08 zzzeek there's no "row" that we would have before invoking a statement
18:29:12 melwitt thanks zzzeek++
18:29:18 melwitt haha yeah. makes sense
18:32:26 melwitt zzzeek: I'm investigating a gate bug (and we enabled connection_debug=100) and noticing that in the failure cases, we get no rows logged for our query for compute nodes. the gate environment on the nodes where it fails is known to be running on nodes with restricted disk iops. do you have any idea what could be happening or how we could tune to handle this environment better?
18:33:05 melwitt like, in what scenarios could we make a query to the db server and then get no response within 60 seconds?
18:33:26 zzzeek melwitt: well eventlet can do that, if you're using eventlet in some case
18:33:43 zzzeek melwitt: another is, the query has a huge cartesian product and is taking too long to ORDER BY
18:33:52 melwitt and if there's any rule of thumb way to deal with that, assuming that the disks cannot be made faster or allow more iops
18:34:14 melwitt we are using eventlet to manage the 60 second client side timeout waiting for the result
18:35:04 zzzeek the eventlet issue is that if some other greenlets are hogging the CPU then the greenlet in question might not be able to get a result back, but this has never been observed at the scale of 60 seconds
18:35:33 zzzeek if the "restricted disk iops" is making MySQL chug to a halt, that would be a thing
18:36:25 zzzeek is that both read and write ops?
18:37:33 melwitt hm, not sure. I'm referencing this ML thread http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505.html
18:39:21 zzzeek so...the issue is, MySQL is actually being observed to be overworked in this case and we want to tune it for reduced disk IO ?

Earlier   Later