Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-13
16:51:10 sean-k-mooney and sice we cant do cross extra spec validation in your validation propsoeal i also prefer that form the avlidation point of view
16:51:36 sean-k-mooney let me know when you want me to review and or play around with it
16:51:47 stephenfin will do
16:51:47 sean-k-mooney im going to drop soon just an fyi
16:51:58 stephenfin ah yeah, it'll be next week anyway
17:16:39 openstackgerrit Elod Illes proposed openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033
17:19:16 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
17:21:15 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036
17:29:09 openstackgerrit Merged openstack/nova stable/rocky: Remove global state from the FakeDriver https://review.opendev.org/712751
17:29:16 melwitt lyarwood: yay, thanks for looking at that. is the same thing happening with the intermittent failures on master?
17:29:56 melwitt *intermittent nova-live-migration failures
17:30:39 lyarwood melwitt: I don't recall seeing this on master but let me have a quick look in logstash
17:31:25 melwitt lyarwood: ok, was just wondering if you knew off the top of your head. I've seen occasional failures of nova-live-migration on master and just curious if it's the same thing. I hope so, cause that would mean your fix would fix that too
17:37:12 lyarwood melwitt: yeah, I can't see anything on master but my elasticsearch foo is awful
17:37:35 melwitt ok np
17:39:38 lyarwood melwitt: FWIW your https://review.opendev.org/#/c/712226/ change failed because of http://status.openstack.org/elastic-recheck/#1813789
17:39:51 lyarwood melwitt: I've not had time to look into that but I have been seeing that across master
17:40:30 melwitt lyarwood: ah thanks. I've been looking at http://status.openstack.org/elastic-recheck/#1844929 again yesterday and today
17:41:06 melwitt so far, no dice
17:42:52 lyarwood melwitt: kk, these all appear rather tricky
17:43:14 melwitt yeah :(
18:28:10 melwitt zzzeek: could you pls sanity check me on this -- this logging is done after getting a response back from the database server right? it's not a client side logging before making the query https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/engine/result.py#L1579
18:28:54 zzzeek melwitt: row logging is after we've executed the statement and we've received rows back from the DBAPI cursor, that line logs the row itself
18:29:08 zzzeek there's no "row" that we would have before invoking a statement
18:29:12 melwitt thanks zzzeek++
18:29:18 melwitt haha yeah. makes sense
18:32:26 melwitt zzzeek: I'm investigating a gate bug (and we enabled connection_debug=100) and noticing that in the failure cases, we get no rows logged for our query for compute nodes. the gate environment on the nodes where it fails is known to be running on nodes with restricted disk iops. do you have any idea what could be happening or how we could tune to handle this environment better?
18:33:05 melwitt like, in what scenarios could we make a query to the db server and then get no response within 60 seconds?
18:33:26 zzzeek melwitt: well eventlet can do that, if you're using eventlet in some case
18:33:43 zzzeek melwitt: another is, the query has a huge cartesian product and is taking too long to ORDER BY
18:33:52 melwitt and if there's any rule of thumb way to deal with that, assuming that the disks cannot be made faster or allow more iops
18:34:14 melwitt we are using eventlet to manage the 60 second client side timeout waiting for the result
18:35:04 zzzeek the eventlet issue is that if some other greenlets are hogging the CPU then the greenlet in question might not be able to get a result back, but this has never been observed at the scale of 60 seconds
18:35:33 zzzeek if the "restricted disk iops" is making MySQL chug to a halt, that would be a thing
18:36:25 zzzeek is that both read and write ops?
18:37:33 melwitt hm, not sure. I'm referencing this ML thread http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505.html
18:39:21 zzzeek so...the issue is, MySQL is actually being observed to be overworked in this case and we want to tune it for reduced disk IO ?
18:40:09 melwitt I was wondering that as a possibility, yes
18:40:48 melwitt I'm playing around with some innodb_ settings in devstack to experiment, but mostly don't know what to try
18:41:42 melwitt I wondered if tuning for that might enable it to work under the reduced disk IO condition
18:53:28 melwitt zzzeek: I have to run for a bit, but if you think of any good tunables I could try in my.conf, pls give me a shout
18:53:55 zzzeek sure...there are some write flags but read is tricky
18:54:31 melwitt *my.cnf
20:13:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036
20:27:24 sean-k-mooney melwitt: how big is the db
20:28:40 sean-k-mooney it would be a bit of a hack but it might be posible to mount its data on tempfs or otherwise use some form of filesystem or block level caching to mask the io limitations
20:34:02 melwitt sean-k-mooney: how can I tell how big is the db? I have been using this page to get an idea of things to try https://dev.mysql.com/doc/refman/5.6/en/optimizing-innodb-diskio.html
20:34:32 sean-k-mooney i was just thining of useing du
20:36:10 melwitt oh, lemme see, there's df output on the job runs
20:37:08 melwitt https://zuul.opendev.org/t/openstack/build/8c91fd21815148d9894ac2bf60893a9e/log/logs/df.txt
20:37:39 sean-k-mooney ya that unfortunetly is not going to show us
20:37:53 sean-k-mooney ill check my local devstack
20:38:27 sean-k-mooney so im seeing about 180mb
20:38:46 sean-k-mooney in /var/lib/mysql
20:38:54 sean-k-mooney so we could just try mounting that on tempfs
20:39:06 sean-k-mooney i think we would have enough ram to do that
20:39:28 melwitt oh, thanks
20:40:22 sean-k-mooney runing a db on tempfs is normaly really dumb but its a ci so ...
20:40:37 sean-k-mooney its not like we care if the data goes away after a reboot
20:40:40 melwitt heh
21:40:31 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
21:41:15 lyarwood melwitt / sean-k-mooney ; ^ if you're around, thoughts on that would be appreciated. Appears to fix the issues we've been seeing on stable/pike.
21:42:29 melwitt cool, will look. thanks for digging into that
21:47:35 lyarwood thanks :)
22:23:41 sean-k-mooney lyarwood: oh is that why the migration job was failing. it was not waiting for ceph after it reconfigred and restarted n-cpu
#openstack-nova - 2020-03-14
03:32:18 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
03:32:18 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
04:39:59 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651
09:51:49 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
12:33:03 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:09:29 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:13:01 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
14:43:07 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
15:37:43 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:38:50 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP pre-commit: Add bashate https://review.opendev.org/713102
16:38:51 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP tox: Add bashate https://review.opendev.org/713103
17:03:44 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
18:35:00 openstackgerrit Merged openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033
19:01:14 openstackgerrit Merged openstack/nova stable/rocky: Avoid circular reference during serialization https://review.opendev.org/709798
21:04:39 openstackgerrit Merged openstack/nova stable/rocky: Fix hypervisors paginted collection_name. https://review.opendev.org/711670
22:39:25 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
22:39:25 openstackgerrit melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116
22:40:50 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
22:40:50 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
#openstack-nova - 2020-03-15
11:45:05 openstackgerrit Vlad Gusev proposed openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132
12:13:42 openstackgerrit Brin Zhang proposed openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428
12:13:43 openstackgerrit Brin Zhang proposed openstack/nova master: Add instance actions v283 samples test https://review.opendev.org/706251
12:13:43 openstackgerrit Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430
13:45:06 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
14:10:53 donnyd is there a place in the meta-data service that would tell you what region an instance was launched in?
15:13:21 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
15:38:34 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:36:16 openstackgerrit Victor Coutellier proposed openstack/nova master: Non-Admin user can filter their instances by more filters https://review.opendev.org/701609
16:40:26 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:41:15 lyarwood ^ set -o xtrace lyarwood--
21:04:30 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
21:14:41 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
#openstack-nova - 2020-03-16
01:00:40 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470
01:03:25 brinzhang gmann: done, thanks. And I replied your question, you can review again while your are free.
02:47:52 openstackgerrit Brin Zhang proposed openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428

Earlier   Later