Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-13
18:33:05 melwitt like, in what scenarios could we make a query to the db server and then get no response within 60 seconds?
18:33:26 zzzeek melwitt: well eventlet can do that, if you're using eventlet in some case
18:33:43 zzzeek melwitt: another is, the query has a huge cartesian product and is taking too long to ORDER BY
18:33:52 melwitt and if there's any rule of thumb way to deal with that, assuming that the disks cannot be made faster or allow more iops
18:34:14 melwitt we are using eventlet to manage the 60 second client side timeout waiting for the result
18:35:04 zzzeek the eventlet issue is that if some other greenlets are hogging the CPU then the greenlet in question might not be able to get a result back, but this has never been observed at the scale of 60 seconds
18:35:33 zzzeek if the "restricted disk iops" is making MySQL chug to a halt, that would be a thing
18:36:25 zzzeek is that both read and write ops?
18:37:33 melwitt hm, not sure. I'm referencing this ML thread http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505.html
18:39:21 zzzeek so...the issue is, MySQL is actually being observed to be overworked in this case and we want to tune it for reduced disk IO ?
18:40:09 melwitt I was wondering that as a possibility, yes
18:40:48 melwitt I'm playing around with some innodb_ settings in devstack to experiment, but mostly don't know what to try
18:41:42 melwitt I wondered if tuning for that might enable it to work under the reduced disk IO condition
18:53:28 melwitt zzzeek: I have to run for a bit, but if you think of any good tunables I could try in my.conf, pls give me a shout
18:53:55 zzzeek sure...there are some write flags but read is tricky
18:54:31 melwitt *my.cnf
20:13:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036
20:27:24 sean-k-mooney melwitt: how big is the db
20:28:40 sean-k-mooney it would be a bit of a hack but it might be posible to mount its data on tempfs or otherwise use some form of filesystem or block level caching to mask the io limitations
20:34:02 melwitt sean-k-mooney: how can I tell how big is the db? I have been using this page to get an idea of things to try https://dev.mysql.com/doc/refman/5.6/en/optimizing-innodb-diskio.html
20:34:32 sean-k-mooney i was just thining of useing du
20:36:10 melwitt oh, lemme see, there's df output on the job runs
20:37:08 melwitt https://zuul.opendev.org/t/openstack/build/8c91fd21815148d9894ac2bf60893a9e/log/logs/df.txt
20:37:39 sean-k-mooney ya that unfortunetly is not going to show us
20:37:53 sean-k-mooney ill check my local devstack
20:38:27 sean-k-mooney so im seeing about 180mb
20:38:46 sean-k-mooney in /var/lib/mysql
20:38:54 sean-k-mooney so we could just try mounting that on tempfs
20:39:06 sean-k-mooney i think we would have enough ram to do that
20:39:28 melwitt oh, thanks
20:40:22 sean-k-mooney runing a db on tempfs is normaly really dumb but its a ci so ...
20:40:37 sean-k-mooney its not like we care if the data goes away after a reboot
20:40:40 melwitt heh
21:40:31 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
21:41:15 lyarwood melwitt / sean-k-mooney ; ^ if you're around, thoughts on that would be appreciated. Appears to fix the issues we've been seeing on stable/pike.
21:42:29 melwitt cool, will look. thanks for digging into that
21:47:35 lyarwood thanks :)
22:23:41 sean-k-mooney lyarwood: oh is that why the migration job was failing. it was not waiting for ceph after it reconfigred and restarted n-cpu
#openstack-nova - 2020-03-14
03:32:18 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
03:32:18 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
04:39:59 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651
09:51:49 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
12:33:03 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:09:29 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:13:01 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
14:43:07 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
15:37:43 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:38:50 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP pre-commit: Add bashate https://review.opendev.org/713102
16:38:51 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP tox: Add bashate https://review.opendev.org/713103
17:03:44 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
18:35:00 openstackgerrit Merged openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033
19:01:14 openstackgerrit Merged openstack/nova stable/rocky: Avoid circular reference during serialization https://review.opendev.org/709798
21:04:39 openstackgerrit Merged openstack/nova stable/rocky: Fix hypervisors paginted collection_name. https://review.opendev.org/711670
22:39:25 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
22:39:25 openstackgerrit melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116
22:40:50 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
22:40:50 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
#openstack-nova - 2020-03-15
11:45:05 openstackgerrit Vlad Gusev proposed openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132
12:13:42 openstackgerrit Brin Zhang proposed openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428
12:13:43 openstackgerrit Brin Zhang proposed openstack/nova master: Add instance actions v283 samples test https://review.opendev.org/706251
12:13:43 openstackgerrit Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430
13:45:06 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
14:10:53 donnyd is there a place in the meta-data service that would tell you what region an instance was launched in?
15:13:21 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
15:38:34 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:36:16 openstackgerrit Victor Coutellier proposed openstack/nova master: Non-Admin user can filter their instances by more filters https://review.opendev.org/701609
16:40:26 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:41:15 lyarwood ^ set -o xtrace lyarwood--
21:04:30 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
21:14:41 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
#openstack-nova - 2020-03-16
01:00:40 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470
01:03:25 brinzhang gmann: done, thanks. And I replied your question, you can review again while your are free.
02:47:52 openstackgerrit Brin Zhang proposed openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428
02:47:53 openstackgerrit Brin Zhang proposed openstack/nova master: Add instance actions v283 samples test https://review.opendev.org/706251
02:47:53 openstackgerrit Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430
03:14:34 openstackgerrit Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494
05:31:05 openstackgerrit Kevin Zhao proposed openstack/nova master: fix ut error on arm64 https://review.opendev.org/713163
05:42:17 openstackgerrit Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494
07:54:56 openstackgerrit Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494
08:56:53 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
09:02:45 brinzhang_ stephenfin: can you review os-volume-attachments refresh default policy patch (the end)? https://review.opendev.org/#/c/710190/ it prevent my destroy-instance-with-datavolume add the completed implementation
09:03:47 brinzhang_ stephenfin: as the same as os-instance-actions policies https://review.opendev.org/#/c/706470/, it prevents the action-event-fault-details feature
09:03:55 lyarwood https://zuul.opendev.org/t/openstack/build/790d84d24ac946ec90712585a301ba3f/log/job-output.txt#6750 <- anyone with more bash-foo than me able to tell me why ceph.sh is exiting with 1 here? This works locally and I can't see anything useful dumped with set -oxtrace
09:04:32 brinzhang_ johnthetubaguy: also need you check
09:04:53 brinzhang_ johnthetubaguy: stephenfin: thanks
09:06:46 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
09:45:49 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Move qemu-img info calls into privsep https://review.opendev.org/706897
09:45:50 openstackgerrit Lee Yarwood proposed openstack/nova master: virt: Pass request context to extend_volume https://review.opendev.org/706899
09:45:50 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Allow the output format of qemu-img info to be controlled https://review.opendev.org/706898
09:45:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use oslo.utils >= 4.1.0 to fetch format-specific image data https://review.opendev.org/710785
09:45:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Correctly resize encrypted LUKSv1 volumes https://review.opendev.org/706900
09:45:52 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Remove Libvirt specific configurable use from qemu_img_info https://review.opendev.org/707591
09:45:52 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Always provide the size in bytes when calling virDomainBlockResize https://review.opendev.org/707590
09:45:53 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Make JSON the default output format of calls to qemu-img info https://review.opendev.org/711679
09:45:53 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove QEMU_VERSION_REQ_SHARED https://review.opendev.org/710239
10:07:38 openstack Launchpad bug 1864020 in OpenStack Compute (nova) "libvirt.libvirtError: Requested operation is not valid: format of backing image %s of image %s was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)" [Undecided,Fix committed] - Assigned to Lee Yarwood (lyarwood)
10:07:38 lyarwood hmmm do I need to be part of a particular group to triage bugs? https://bugs.launchpad.net/nova/+bug/1864020
10:07:49 lyarwood I thought I was but I can't seem to set the importance of the above bug
10:09:38 donnyd nightmare_unreal: check this out https://docs.openstack.org/nova/train/contributor/index.html
10:09:52 nightmare_unreal sure

Earlier   Later