| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-13 | |||
| 16:47:43 | stephenfin | because I don't want to force that 'hw:cpu_dedicated_mask' | |
| 16:47:54 | stephenfin | and it would be nice for them to behave similarly | |
| 16:48:05 | sean-k-mooney | stephenfin: ya i would not mind doing that | |
| 16:48:23 | sean-k-mooney | stephenfin: we said the dedicated mask should follow the rules for the config opntion | |
| 16:48:30 | sean-k-mooney | not the rules for the realtime mask | |
| 16:48:33 | stephenfin | the only reason I see to not do that is jaypipes wanted us to kill 'hw:cpu_realtime_mask' in favor of 'hw:cpu_realtime_set' | |
| 16:48:42 | stephenfin | but tbh, I don't think it's worth the effort | |
| 16:48:55 | stephenfin | we haven't deprecated flavor extra specs before. I don't even want to get into that | |
| 16:49:06 | sean-k-mooney | i would be ok with that i guess but yat that ^ | |
| 16:49:10 | stephenfin | ditto for image metadata props, for that matter | |
| 16:49:33 | stephenfin | okay, sweet | |
| 16:49:40 | stephenfin | I can revive those patches so | |
| 16:49:43 | stephenfin | whoo, rebase fun! | |
| 16:50:31 | sean-k-mooney | stephenfin: cool. i prefer each option to do one thing and one thing only. but do what you think is best | |
| 16:50:42 | sean-k-mooney | its a preference not a blocker for me | |
| 16:51:10 | sean-k-mooney | and sice we cant do cross extra spec validation in your validation propsoeal i also prefer that form the avlidation point of view | |
| 16:51:36 | sean-k-mooney | let me know when you want me to review and or play around with it | |
| 16:51:47 | stephenfin | will do | |
| 16:51:47 | sean-k-mooney | im going to drop soon just an fyi | |
| 16:51:58 | stephenfin | ah yeah, it'll be next week anyway | |
| 17:16:39 | openstackgerrit | Elod Illes proposed openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033 | |
| 17:19:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | |
| 17:21:15 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036 | |
| 17:29:09 | openstackgerrit | Merged openstack/nova stable/rocky: Remove global state from the FakeDriver https://review.opendev.org/712751 | |
| 17:29:16 | melwitt | lyarwood: yay, thanks for looking at that. is the same thing happening with the intermittent failures on master? | |
| 17:29:56 | melwitt | *intermittent nova-live-migration failures | |
| 17:30:39 | lyarwood | melwitt: I don't recall seeing this on master but let me have a quick look in logstash | |
| 17:31:25 | melwitt | lyarwood: ok, was just wondering if you knew off the top of your head. I've seen occasional failures of nova-live-migration on master and just curious if it's the same thing. I hope so, cause that would mean your fix would fix that too | |
| 17:37:12 | lyarwood | melwitt: yeah, I can't see anything on master but my elasticsearch foo is awful | |
| 17:37:35 | melwitt | ok np | |
| 17:39:38 | lyarwood | melwitt: FWIW your https://review.opendev.org/#/c/712226/ change failed because of http://status.openstack.org/elastic-recheck/#1813789 | |
| 17:39:51 | lyarwood | melwitt: I've not had time to look into that but I have been seeing that across master | |
| 17:40:30 | melwitt | lyarwood: ah thanks. I've been looking at http://status.openstack.org/elastic-recheck/#1844929 again yesterday and today | |
| 17:41:06 | melwitt | so far, no dice | |
| 17:42:52 | lyarwood | melwitt: kk, these all appear rather tricky | |
| 17:43:14 | melwitt | yeah :( | |
| 18:28:10 | melwitt | zzzeek: could you pls sanity check me on this -- this logging is done after getting a response back from the database server right? it's not a client side logging before making the query https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/engine/result.py#L1579 | |
| 18:28:54 | zzzeek | melwitt: row logging is after we've executed the statement and we've received rows back from the DBAPI cursor, that line logs the row itself | |
| 18:29:08 | zzzeek | there's no "row" that we would have before invoking a statement | |
| 18:29:12 | melwitt | thanks zzzeek++ | |
| 18:29:18 | melwitt | haha yeah. makes sense | |
| 18:32:26 | melwitt | zzzeek: I'm investigating a gate bug (and we enabled connection_debug=100) and noticing that in the failure cases, we get no rows logged for our query for compute nodes. the gate environment on the nodes where it fails is known to be running on nodes with restricted disk iops. do you have any idea what could be happening or how we could tune to handle this environment better? | |
| 18:33:05 | melwitt | like, in what scenarios could we make a query to the db server and then get no response within 60 seconds? | |
| 18:33:26 | zzzeek | melwitt: well eventlet can do that, if you're using eventlet in some case | |
| 18:33:43 | zzzeek | melwitt: another is, the query has a huge cartesian product and is taking too long to ORDER BY | |
| 18:33:52 | melwitt | and if there's any rule of thumb way to deal with that, assuming that the disks cannot be made faster or allow more iops | |
| 18:34:14 | melwitt | we are using eventlet to manage the 60 second client side timeout waiting for the result | |
| 18:35:04 | zzzeek | the eventlet issue is that if some other greenlets are hogging the CPU then the greenlet in question might not be able to get a result back, but this has never been observed at the scale of 60 seconds | |
| 18:35:33 | zzzeek | if the "restricted disk iops" is making MySQL chug to a halt, that would be a thing | |
| 18:36:25 | zzzeek | is that both read and write ops? | |
| 18:37:33 | melwitt | hm, not sure. I'm referencing this ML thread http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505.html | |
| 18:39:21 | zzzeek | so...the issue is, MySQL is actually being observed to be overworked in this case and we want to tune it for reduced disk IO ? | |
| 18:40:09 | melwitt | I was wondering that as a possibility, yes | |
| 18:40:48 | melwitt | I'm playing around with some innodb_ settings in devstack to experiment, but mostly don't know what to try | |
| 18:41:42 | melwitt | I wondered if tuning for that might enable it to work under the reduced disk IO condition | |
| 18:53:28 | melwitt | zzzeek: I have to run for a bit, but if you think of any good tunables I could try in my.conf, pls give me a shout | |
| 18:53:55 | zzzeek | sure...there are some write flags but read is tricky | |
| 18:54:31 | melwitt | *my.cnf | |
| 20:13:46 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036 | |
| 20:27:24 | sean-k-mooney | melwitt: how big is the db | |
| 20:28:40 | sean-k-mooney | it would be a bit of a hack but it might be posible to mount its data on tempfs or otherwise use some form of filesystem or block level caching to mask the io limitations | |
| 20:34:02 | melwitt | sean-k-mooney: how can I tell how big is the db? I have been using this page to get an idea of things to try https://dev.mysql.com/doc/refman/5.6/en/optimizing-innodb-diskio.html | |
| 20:34:32 | sean-k-mooney | i was just thining of useing du | |
| 20:36:10 | melwitt | oh, lemme see, there's df output on the job runs | |
| 20:37:08 | melwitt | https://zuul.opendev.org/t/openstack/build/8c91fd21815148d9894ac2bf60893a9e/log/logs/df.txt | |
| 20:37:39 | sean-k-mooney | ya that unfortunetly is not going to show us | |
| 20:37:53 | sean-k-mooney | ill check my local devstack | |
| 20:38:27 | sean-k-mooney | so im seeing about 180mb | |
| 20:38:46 | sean-k-mooney | in /var/lib/mysql | |
| 20:38:54 | sean-k-mooney | so we could just try mounting that on tempfs | |
| 20:39:06 | sean-k-mooney | i think we would have enough ram to do that | |
| 20:39:28 | melwitt | oh, thanks | |
| 20:40:22 | sean-k-mooney | runing a db on tempfs is normaly really dumb but its a ci so ... | |
| 20:40:37 | sean-k-mooney | its not like we care if the data goes away after a reboot | |
| 20:40:40 | melwitt | heh | |
| 21:40:31 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | |
| 21:41:15 | lyarwood | melwitt / sean-k-mooney ; ^ if you're around, thoughts on that would be appreciated. Appears to fix the issues we've been seeing on stable/pike. | |
| 21:42:29 | melwitt | cool, will look. thanks for digging into that | |
| 21:47:35 | lyarwood | thanks :) | |
| 22:23:41 | sean-k-mooney | lyarwood: oh is that why the migration job was failing. it was not waiting for ceph after it reconfigred and restarted n-cpu | |
| #openstack-nova - 2020-03-14 | |||
| 03:32:18 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 03:32:18 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 04:39:59 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651 | |
| 09:51:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | |
| 12:33:03 | openstackgerrit | Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089 | |
| 13:09:29 | openstackgerrit | Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089 | |
| 13:13:01 | openstackgerrit | Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089 | |
| 14:43:07 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | |
| 15:37:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | |
| 16:38:50 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP pre-commit: Add bashate https://review.opendev.org/713102 | |
| 16:38:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP tox: Add bashate https://review.opendev.org/713103 | |
| 17:03:44 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | |
| 18:35:00 | openstackgerrit | Merged openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033 | |
| 19:01:14 | openstackgerrit | Merged openstack/nova stable/rocky: Avoid circular reference during serialization https://review.opendev.org/709798 | |
| 21:04:39 | openstackgerrit | Merged openstack/nova stable/rocky: Fix hypervisors paginted collection_name. https://review.opendev.org/711670 | |
| 22:39:25 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 22:39:25 | openstackgerrit | melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116 | |
| 22:40:50 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 22:40:50 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| #openstack-nova - 2020-03-15 | |||
| 11:45:05 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132 | |