Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-13
18:32:26 melwitt zzzeek: I'm investigating a gate bug (and we enabled connection_debug=100) and noticing that in the failure cases, we get no rows logged for our query for compute nodes. the gate environment on the nodes where it fails is known to be running on nodes with restricted disk iops. do you have any idea what could be happening or how we could tune to handle this environment better?
18:33:05 melwitt like, in what scenarios could we make a query to the db server and then get no response within 60 seconds?
18:33:26 zzzeek melwitt: well eventlet can do that, if you're using eventlet in some case
18:33:43 zzzeek melwitt: another is, the query has a huge cartesian product and is taking too long to ORDER BY
18:33:52 melwitt and if there's any rule of thumb way to deal with that, assuming that the disks cannot be made faster or allow more iops
18:34:14 melwitt we are using eventlet to manage the 60 second client side timeout waiting for the result
18:35:04 zzzeek the eventlet issue is that if some other greenlets are hogging the CPU then the greenlet in question might not be able to get a result back, but this has never been observed at the scale of 60 seconds
18:35:33 zzzeek if the "restricted disk iops" is making MySQL chug to a halt, that would be a thing
18:36:25 zzzeek is that both read and write ops?
18:37:33 melwitt hm, not sure. I'm referencing this ML thread http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505.html
18:39:21 zzzeek so...the issue is, MySQL is actually being observed to be overworked in this case and we want to tune it for reduced disk IO ?
18:40:09 melwitt I was wondering that as a possibility, yes
18:40:48 melwitt I'm playing around with some innodb_ settings in devstack to experiment, but mostly don't know what to try
18:41:42 melwitt I wondered if tuning for that might enable it to work under the reduced disk IO condition
18:53:28 melwitt zzzeek: I have to run for a bit, but if you think of any good tunables I could try in my.conf, pls give me a shout
18:53:55 zzzeek sure...there are some write flags but read is tricky
18:54:31 melwitt *my.cnf
20:13:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: WIP nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036
20:27:24 sean-k-mooney melwitt: how big is the db
20:28:40 sean-k-mooney it would be a bit of a hack but it might be posible to mount its data on tempfs or otherwise use some form of filesystem or block level caching to mask the io limitations
20:34:02 melwitt sean-k-mooney: how can I tell how big is the db? I have been using this page to get an idea of things to try https://dev.mysql.com/doc/refman/5.6/en/optimizing-innodb-diskio.html
20:34:32 sean-k-mooney i was just thining of useing du
20:36:10 melwitt oh, lemme see, there's df output on the job runs
20:37:08 melwitt https://zuul.opendev.org/t/openstack/build/8c91fd21815148d9894ac2bf60893a9e/log/logs/df.txt
20:37:39 sean-k-mooney ya that unfortunetly is not going to show us
20:37:53 sean-k-mooney ill check my local devstack
20:38:27 sean-k-mooney so im seeing about 180mb
20:38:46 sean-k-mooney in /var/lib/mysql
20:38:54 sean-k-mooney so we could just try mounting that on tempfs
20:39:06 sean-k-mooney i think we would have enough ram to do that
20:39:28 melwitt oh, thanks
20:40:22 sean-k-mooney runing a db on tempfs is normaly really dumb but its a ci so ...
20:40:37 sean-k-mooney its not like we care if the data goes away after a reboot
20:40:40 melwitt heh
21:40:31 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
21:41:15 lyarwood melwitt / sean-k-mooney ; ^ if you're around, thoughts on that would be appreciated. Appears to fix the issues we've been seeing on stable/pike.
21:42:29 melwitt cool, will look. thanks for digging into that
21:47:35 lyarwood thanks :)
22:23:41 sean-k-mooney lyarwood: oh is that why the migration job was failing. it was not waiting for ceph after it reconfigred and restarted n-cpu
#openstack-nova - 2020-03-14
03:32:18 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
03:32:18 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
04:39:59 openstackgerrit Brin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-patch CLI https://review.opendev.org/712651
09:51:49 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
12:33:03 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:09:29 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
13:13:01 openstackgerrit Victor Coutellier proposed openstack/python-novaclient master: Microversion 2.83 - Add more filters for the nova list command https://review.opendev.org/713089
14:43:07 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
15:37:43 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:38:50 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP pre-commit: Add bashate https://review.opendev.org/713102
16:38:51 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP tox: Add bashate https://review.opendev.org/713103
17:03:44 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
18:35:00 openstackgerrit Merged openstack/nova stable/rocky: Enhance service restart in functional env https://review.opendev.org/713033
19:01:14 openstackgerrit Merged openstack/nova stable/rocky: Avoid circular reference during serialization https://review.opendev.org/709798
21:04:39 openstackgerrit Merged openstack/nova stable/rocky: Fix hypervisors paginted collection_name. https://review.opendev.org/711670
22:39:25 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
22:39:25 openstackgerrit melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116
22:40:50 openstack bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929
22:40:50 openstackgerrit melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478
#openstack-nova - 2020-03-15
11:45:05 openstackgerrit Vlad Gusev proposed openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132
12:13:42 openstackgerrit Brin Zhang proposed openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428
12:13:43 openstackgerrit Brin Zhang proposed openstack/nova master: Add instance actions v283 samples test https://review.opendev.org/706251
12:13:43 openstackgerrit Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430
13:45:06 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
14:10:53 donnyd is there a place in the meta-data service that would tell you what region an instance was launched in?
15:13:21 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
15:38:34 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:36:16 openstackgerrit Victor Coutellier proposed openstack/nova master: Non-Admin user can filter their instances by more filters https://review.opendev.org/701609
16:40:26 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
16:41:15 lyarwood ^ set -o xtrace lyarwood--
21:04:30 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
21:14:41 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
#openstack-nova - 2020-03-16
01:00:40 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470
01:03:25 brinzhang gmann: done, thanks. And I replied your question, you can review again while your are free.
02:47:52 openstackgerrit Brin Zhang proposed openstack/nova master: Store instance action event exc_val fault details https://review.opendev.org/694428
02:47:53 openstackgerrit Brin Zhang proposed openstack/nova master: Add instance actions v283 samples test https://review.opendev.org/706251
02:47:53 openstackgerrit Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430
03:14:34 openstackgerrit Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494
05:31:05 openstackgerrit Kevin Zhao proposed openstack/nova master: fix ut error on arm64 https://review.opendev.org/713163
05:42:17 openstackgerrit Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494
07:54:56 openstackgerrit Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494
08:56:53 openstackgerrit waleed mousa proposed openstack/os-vif master: [Follow Up] OVS DPDK port representors support https://review.opendev.org/705018
09:02:45 brinzhang_ stephenfin: can you review os-volume-attachments refresh default policy patch (the end)? https://review.opendev.org/#/c/710190/ it prevent my destroy-instance-with-datavolume add the completed implementation
09:03:47 brinzhang_ stephenfin: as the same as os-instance-actions policies https://review.opendev.org/#/c/706470/, it prevents the action-event-fault-details feature
09:03:55 lyarwood https://zuul.opendev.org/t/openstack/build/790d84d24ac946ec90712585a301ba3f/log/job-output.txt#6750 <- anyone with more bash-foo than me able to tell me why ceph.sh is exiting with 1 here? This works locally and I can't see anything useful dumped with set -oxtrace
09:04:32 brinzhang_ johnthetubaguy: also need you check
09:04:53 brinzhang_ johnthetubaguy: stephenfin: thanks
09:06:46 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035
09:45:49 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Move qemu-img info calls into privsep https://review.opendev.org/706897
09:45:50 openstackgerrit Lee Yarwood proposed openstack/nova master: virt: Pass request context to extend_volume https://review.opendev.org/706899
09:45:50 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Allow the output format of qemu-img info to be controlled https://review.opendev.org/706898
09:45:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use oslo.utils >= 4.1.0 to fetch format-specific image data https://review.opendev.org/710785
09:45:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Correctly resize encrypted LUKSv1 volumes https://review.opendev.org/706900
09:45:52 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Remove Libvirt specific configurable use from qemu_img_info https://review.opendev.org/707591
09:45:52 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Always provide the size in bytes when calling virDomainBlockResize https://review.opendev.org/707590
09:45:53 openstackgerrit Lee Yarwood proposed openstack/nova master: images: Make JSON the default output format of calls to qemu-img info https://review.opendev.org/711679
09:45:53 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove QEMU_VERSION_REQ_SHARED https://review.opendev.org/710239
10:07:38 openstack Launchpad bug 1864020 in OpenStack Compute (nova) "libvirt.libvirtError: Requested operation is not valid: format of backing image %s of image %s was not specified in the image metadata (See https://libvirt.org/kbase/backing_chains.html for troubleshooting)" [Undecided,Fix committed] - Assigned to Lee Yarwood (lyarwood)
10:07:38 lyarwood hmmm do I need to be part of a particular group to triage bugs? https://bugs.launchpad.net/nova/+bug/1864020
10:07:49 lyarwood I thought I was but I can't seem to set the importance of the above bug
10:09:38 donnyd nightmare_unreal: check this out https://docs.openstack.org/nova/train/contributor/index.html

Earlier   Later