Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-13
16:44:33 stephenfin having to set both hw:cpu_dedicated_mask and hw:cpu_realtime_mask
16:44:46 sean-k-mooney right so i would make the cpu_realtime_mask optional
16:44:50 stephenfin if you want a real-time instance with the non-realtime cores floating
16:44:55 sean-k-mooney and always require the cpu_dedicated_mask
16:45:18 stephenfin but then you have a difference of behavior elsewhere
16:45:28 sean-k-mooney and make it so that if you dont set cpu_realtime_mask but do set hw:cpu_realtime=true then we use the hw:dedciated_cpu_mask
16:45:41 stephenfin now you have to use 'hw:cpu_realtime_mask' when using dedicated
16:45:52 stephenfin but use 'hw:cpu_dedicated_mask' when using mixed
16:46:48 stephenfin anyway, we'll invariably debate this in the patch so back to my original question
16:46:52 sean-k-mooney i think its a much simpler rule to say if you want realtime always use realtime_mask and if you want mixed always use the dedicated_mask
16:47:18 sean-k-mooney if you want both use both and require the realtime mask must be a subset of the dedicated mask
16:47:32 stephenfin if we're doing this, do we want to fix that annoying thing where hw:cpu_realtime_mask has to be preceded by carat?
16:47:43 stephenfin because I don't want to force that 'hw:cpu_dedicated_mask'
16:47:54 stephenfin and it would be nice for them to behave similarly
16:48:05 sean-k-mooney stephenfin: ya i would not mind doing that
16:48:23 sean-k-mooney stephenfin: we said the dedicated mask should follow the rules for the config opntion
16:48:30 sean-k-mooney not the rules for the realtime mask
16:48:33 stephenfin the only reason I see to not do that is jaypipes wanted us to kill 'hw:cpu_realtime_mask' in favor of 'hw:cpu_realtime_set'
16:48:42 stephenfin but tbh, I don't think it's worth the effort
16:48:55 stephenfin we haven't deprecated flavor extra specs before. I don't even want to get into that
16:49:06 sean-k-mooney i would be ok with that i guess but yat that ^
16:49:10 stephenfin ditto for image metadata props, for that matter
16:49:33 stephenfin okay, sweet
16:49:40 stephenfin I can revive those patches so
16:49:43 stephenfin whoo, rebase fun!
16:50:31 sean-k-mooney stephenfin: cool. i prefer each option to do one thing and one thing only. but do what you think is best
16:50:42 sean-k-mooney its a preference not a blocker for me
16:51:10 sean-k-mooney and sice we cant do cross extra spec validation in your validation propsoeal i also prefer that form the avlidation point of view
16:51:36 sean-k-mooney let me know when you want me to review and or play around with it
16:51:47 stephenfin will do
16:51:47 sean-k-mooney im going to drop soon just an fyi
16:51:58 stephenfin ah yeah, it'll be next week anyway
17:16:39 openstackgerrit Elod Illes proposed openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033
17:19:16 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
17:21:15 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036
17:29:09 openstackgerrit Merged openstack/nova stable/rocky: Remove global state from the FakeDriver https://review.opendev.org/712751
17:29:16 melwitt lyarwood: yay, thanks for looking at that. is the same thing happening with the intermittent failures on master?
17:29:56 melwitt *intermittent nova-live-migration failures
17:30:39 lyarwood melwitt: I don't recall seeing this on master but let me have a quick look in logstash
17:31:25 melwitt lyarwood: ok, was just wondering if you knew off the top of your head. I've seen occasional failures of nova-live-migration on master and just curious if it's the same thing. I hope so, cause that would mean your fix would fix that too
17:37:12 lyarwood melwitt: yeah, I can't see anything on master but my elasticsearch foo is awful
17:37:35 melwitt ok np
17:39:38 lyarwood melwitt: FWIW your https://review.opendev.org/#/c/712226/ change failed because of http://status.openstack.org/elastic-recheck/#1813789
17:39:51 lyarwood melwitt: I've not had time to look into that but I have been seeing that across master
17:40:30 melwitt lyarwood: ah thanks. I've been looking at http://status.openstack.org/elastic-recheck/#1844929 again yesterday and today
17:41:06 melwitt so far, no dice
17:42:52 lyarwood melwitt: kk, these all appear rather tricky
17:43:14 melwitt yeah :(
18:28:10 melwitt zzzeek: could you pls sanity check me on this -- this logging is done after getting a response back from the database server right? it's not a client side logging before making the query https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/engine/result.py#L1579
18:28:54 zzzeek melwitt: row logging is after we've executed the statement and we've received rows back from the DBAPI cursor, that line logs the row itself
18:29:08 zzzeek there's no "row" that we would have before invoking a statement
18:29:12 melwitt thanks zzzeek++
18:29:18 melwitt haha yeah. makes sense
18:32:26 melwitt zzzeek: I'm investigating a gate bug (and we enabled connection_debug=100) and noticing that in the failure cases, we get no rows logged for our query for compute nodes. the gate environment on the nodes where it fails is known to be running on nodes with restricted disk iops. do you have any idea what could be happening or how we could tune to handle this environment better?
18:33:05 melwitt like, in what scenarios could we make a query to the db server and then get no response within 60 seconds?
18:33:26 zzzeek melwitt: well eventlet can do that, if you're using eventlet in some case
18:33:43 zzzeek melwitt: another is, the query has a huge cartesian product and is taking too long to ORDER BY
18:33:52 melwitt and if there's any rule of thumb way to deal with that, assuming that the disks cannot be made faster or allow more iops
18:34:14 melwitt we are using eventlet to manage the 60 second client side timeout waiting for the result
18:35:04 zzzeek the eventlet issue is that if some other greenlets are hogging the CPU then the greenlet in question might not be able to get a result back, but this has never been observed at the scale of 60 seconds
18:35:33 zzzeek if the "restricted disk iops" is making MySQL chug to a halt, that would be a thing
18:36:25 zzzeek is that both read and write ops?
18:37:33 melwitt hm, not sure. I'm referencing this ML thread http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505.html
18:39:21 zzzeek so...the issue is, MySQL is actually being observed to be overworked in this case and we want to tune it for reduced disk IO ?
18:40:09 melwitt I was wondering that as a possibility, yes
18:40:48 melwitt I'm playing around with some innodb_ settings in devstack to experiment, but mostly don't know what to try
18:41:42 melwitt I wondered if tuning for that might enable it to work under the reduced disk IO condition
18:53:28 melwitt zzzeek: I have to run for a bit, but if you think of any good tunables I could try in my.conf, pls give me a shout
18:53:55 zzzeek sure...there are some write flags but read is tricky
18:54:31 melwitt *my.cnf
20:13:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036
20:27:24 sean-k-mooney melwitt: how big is the db
20:28:40 sean-k-mooney it would be a bit of a hack but it might be posible to mount its data on tempfs or otherwise use some form of filesystem or block level caching to mask the io limitations
20:34:02 melwitt sean-k-mooney: how can I tell how big is the db? I have been using this page to get an idea of things to try https://dev.mysql.com/doc/refman/5.6/en/optimizing-innodb-diskio.html
20:34:32 sean-k-mooney i was just thining of useing du
20:36:10 melwitt oh, lemme see, there's df output on the job runs
20:37:08 melwitt https://zuul.opendev.org/t/openstack/build/8c91fd21815148d9894ac2bf60893a9e/log/logs/df.txt
20:37:39 sean-k-mooney ya that unfortunetly is not going to show us
20:37:53 sean-k-mooney ill check my local devstack
20:38:27 sean-k-mooney so im seeing about 180mb
20:38:46 sean-k-mooney in /var/lib/mysql
20:38:54 sean-k-mooney so we could just try mounting that on tempfs
20:39:06 sean-k-mooney i think we would have enough ram to do that
20:39:28 melwitt oh, thanks
20:40:22 sean-k-mooney runing a db on tempfs is normaly really dumb but its a ci so ...
20:40:37 sean-k-mooney its not like we care if the data goes away after a reboot
20:40:40 melwitt heh
21:40:31 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
21:41:15 lyarwood melwitt / sean-k-mooney ; ^ if you're around, thoughts on that would be appreciated. Appears to fix the issues we've been seeing on stable/pike.
21:42:29 melwitt cool, will look. thanks for digging into that
21:47:35 lyarwood thanks :)
22:23:41 sean-k-mooney lyarwood: oh is that why the migration job was failing. it was not waiting for ceph after it reconfigred and restarted n-cpu
#openstack-nova - 2020-03-14
03:32:18 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
03:32:18 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
04:39:59 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651
09:51:49 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
12:33:03 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:09:29 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:13:01 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
14:43:07 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035

Earlier   Later