| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-29 | |||
| 16:40:40 | aspiers | seems completely unrelated to my patch which is failing in the gate | |
| 16:40:50 | aspiers | and I think I've seen it a few times in the last 24 hours | |
| 16:43:00 | aspiers | oh, that's just a warning | |
| 16:43:34 | aspiers | still trying to find the real issue in http://logs.openstack.org/66/661266/4/gate/openstack-tox-lower-constraints/7af4be7/job-output.txt.gz | |
| 16:43:42 | aspiers | I see a bunch of timeouts | |
| 16:45:26 | openstackgerrit | Merged openstack/os-vif stable/stein: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655678 | |
| 16:47:06 | kashyap | aspiers: Remind me again, in 'os-traits' repo, what to do when 'tox' complains too long lines for a URL? | |
| 16:47:23 | aspiers | kashyap: isn't that just a warning? | |
| 16:47:36 | aspiers | if not, I guess # noqa or similar | |
| 16:48:05 | kashyap | aspiers: It moans an ERROR -- "E501 line too long (108 > 79 characters)" | |
| 16:48:28 | kashyap | aspiers: Before I go off shortly, I'm pushing the change. For convenience, I'm including a short "changelog" noting explicitly what I addressed | |
| 16:48:55 | kashyap | ... so that if you want to rebase on it tomm. | |
| 16:49:01 | aspiers | kashyap: http://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html#in-line-ignoring-errors | |
| 16:49:09 | aspiers | # noqa: E501 | |
| 16:49:42 | kashyap | Right, thanks! | |
| 16:53:52 | mriedem | aspiers: that duplicate index thing is super old yes | |
| 17:05:10 | aspiers | mriedem: the failures are very similar to https://bugs.launchpad.net/nova/+bug/1823251, but without the InterfaceError | |
| 17:05:11 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 17:06:12 | aspiers | I wonder if the InterfaceError is just one symptom of a timeout | |
| 17:06:24 | aspiers | My timeouts show stuff like IndexError: tuple index out of range | |
| 17:06:46 | aspiers | and 'sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically.' | |
| 17:08:26 | aspiers | ah, that latter one looks similar to https://bugs.launchpad.net/nova/+bug/1793364 | |
| 17:08:28 | openstack | Launchpad bug 1793364 in Manila "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [Undecided,In progress] - Assigned to Goutham Pacha Ravi (gouthamr) | |
| 17:09:00 | mriedem | right | |
| 17:09:05 | mriedem | i've been down that road | |
| 17:10:41 | aspiers | I guess elastic recheck will tell me as much when the other gate jobs finish | |
| 17:11:20 | openstackgerrit | Kashyap Chamarthy proposed openstack/os-traits master: hw: cpu: Rework the directory layout; add missing traits https://review.opendev.org/655193 | |
| 17:12:42 | kashyap | aspiers: When you get a moment, does the short change log address all things you want --^ | |
| 17:12:50 | aspiers | kashyap: already looking at it | |
| 17:13:00 | kashyap | Thank you very much, sir. | |
| 17:14:06 | kashyap | aspiers: Yeah, I wondered, too. Just cargo-culted it. | |
| 17:15:04 | aspiers | +1 | |
| 17:15:50 | kashyap | aspiers: Ah, thanks. My focus was in Gerrit, and I saw the "Update" pop-up (your comment). I was expecting a firm -1 :D | |
| 17:16:02 | aspiers | Oh ye of little faith :) | |
| 17:16:03 | kashyap | Okay, now I cook food | |
| 17:16:07 | aspiers | Good plan | |
| 17:35:27 | ganso | lyarwood, melwitt, dansmith: Hello! When you have a minute, could you please take a look at those backports that are missing just a final +2? https://review.opendev.org/#/c/652153 , https://review.opendev.org/#/c/658136 and https://review.opendev.org/#/c/661349 ? Thanks in advance! =) | |
| 17:35:30 | ganso | tonyb: thanks a lot =D | |
| 17:49:23 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up for counting quota usage from placement https://review.opendev.org/662056 | |
| 18:24:03 | melwitt | mriedem: replied to your comments on https://review.opendev.org/#/c/638073 | |
| 18:24:22 | melwitt | and I dunno if you saw, but it looks like https://review.opendev.org/661391 fixes the novnc tests on the multi-cell job | |
| 18:36:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Run revert resize tests in nova-live-migration https://review.opendev.org/653498 | |
| 18:36:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881 | |
| 18:36:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [DNM] use iptables in nova-multinode https://review.opendev.org/660782 | |
| 18:37:07 | artom | mriedem, ^^ when you get some time, assuming unit tests pass (I've fixed all of them, normally). I think sean-k-mooney and I have finally figured it out. | |
| 18:37:49 | artom | The DNM patch is... optional? I'm not sure what we want to do upstream, but it's good to have it on top to show that both hybrid and normal plug work | |
| 18:47:53 | sean-k-mooney | the DNM patch change the job to use iptables. if we wanted to merge it we should add a second job that use iptable rather then changing multinode job to use it instead of the defualt contrack driver | |
| 18:49:11 | konetzed | will the scheduler honor setting ram_weight_multiplier as extra metadata on a host aggregate? The goal is just to have one host aggregate stack vs spread. | |
| 18:50:52 | sean-k-mooney | konetzed: i dont think so https://github.com/openstack/nova/blob/master/nova/scheduler/weights/ram.py | |
| 18:51:33 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L1089 | |
| 18:51:43 | melwitt | konetzed: to do that, I think you'd have to set ram_weight_multiplier as desired in the nova.conf of the compute hosts that are a member of the host aggregate you want to stack vs spread | |
| 18:51:47 | sean-k-mooney | actully look like it might | |
| 18:52:11 | melwitt | ok, I didn't know that :\ | |
| 18:52:45 | sean-k-mooney | neither did i i assume we broke that when we broke the allocation ratios | |
| 18:52:49 | sean-k-mooney | i guess not | |
| 18:53:42 | sean-k-mooney | its proably suppor fragile if the host is in two aggrates with different values | |
| 18:54:57 | mriedem | you can have per-aggregate weights | |
| 18:55:27 | sean-k-mooney | well from the code yes but if a host is in two aggrates and they have differnt values im not sure what we do | |
| 18:55:43 | konetzed | melwitt: sean-k-mooney: thanks for the quick answer. | |
| 18:55:54 | mriedem | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 18:56:00 | mriedem | "If more than one value is found for a host in aggregate metadata, the minimum value will be used." | |
| 18:56:02 | mriedem | depends on the weigher | |
| 18:56:05 | mriedem | but the docs are all tehre | |
| 18:56:06 | mriedem | *there | |
| 18:56:10 | mriedem | per-aggregate weights is new in stein | |
| 18:56:13 | mriedem | yikun added it | |
| 18:56:27 | melwitt | konetzed ^ updated answer :P sorry | |
| 18:57:12 | konetzed | mriedem: thanks. Looks like i need to update from Rocky ;) | |
| 19:26:34 | mriedem | melwitt: ack i'll get back to it in a bit | |
| 19:26:42 | melwitt | thanks | |
| 19:27:01 | mriedem | unrelated plug for a question about the first stable stein release for anyone that has input http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006617.html | |
| 19:27:11 | mriedem | otherwise i'll probably just punt and un-WIP the release patch | |
| 19:27:52 | sean-k-mooney | mriedem: oh ya i ment to reply. i think adding a known issuse release not makes sense | |
| 19:28:01 | sean-k-mooney | im just not sure what it shoudl say | |
| 19:33:16 | mriedem | btw is this the heartbeat thing we'd expect to see in the n-api logs for that bug http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/screen-n-api.txt.gz?level=TRACE#_May_29_18_00_29_897709 ? | |
| 19:33:25 | mriedem | because it's all over these tempest-pg-full logs | |
| 19:36:15 | melwitt | I haven't actually seen that log for this.. when I reproduced it in devstack I only saw the "connection reset" error message from oslo.messaging. and some other messages about retrying/reconnecting | |
| 19:37:35 | mriedem | i see lots of "missed heartbeats from client, timeout: 60s" in the rabbit logs | |
| 19:37:38 | mriedem | http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/rabbitmq/rabbit@ubuntu-bionic-rax-iad-0006739854_log.txt.gz | |
| 19:37:42 | sean-k-mooney | it looks like the tcp connection is also being closed | |
| 19:38:11 | sean-k-mooney | mriedem: ya that is expected | |
| 19:39:22 | sean-k-mooney | if we dont disabel monkey patching or disabel the heartbeat then we will have missed heartbeat message in rabbitmq | |
| 19:40:41 | mriedem | i guess my point is there was concern about not hitting this in the gate | |
| 19:40:55 | mriedem | even though we run with uwsgi in devstack and eventlet monkey-patched | |
| 19:41:38 | sean-k-mooney | well in the gate we should see this since we have 1 thread per wsgi process | |
| 19:41:48 | sean-k-mooney | but the api sould actuly still work | |
| 19:42:12 | melwitt | yeah, the gate is normally too busy with activity to cause the heartbeat interruption where the wsgi app pauses bc of lack of activity | |
| 19:43:20 | melwitt | so... I dunno how that's showing up in tempest-pg-full :\ | |
| 19:43:40 | zigo | mriedem: Hey matt, thanks again for your patch, it fixed it for us. I've uploaded the fix to Sid, and hopefully it will be part of Buster (currently filling-up an unblock bug to the Debian release team). | |
| 19:44:35 | mriedem | zigo: yw. as noted on the review, i'll likely be modifying that patch a bit. | |
| 19:44:37 | sean-k-mooney | melwitt: wll its not printing constantly so it posible that ocationally we do time out even in the gate | |
| 19:45:17 | melwitt | o | |
| 19:48:52 | mriedem | brb | |
| 19:50:14 | efried | aspiers, kashyap: Minor tweaks to https://review.opendev.org/#/c/655193/ if either of you is around. | |
| 19:50:24 | aspiers | kashyap's gone I think | |
| 19:50:31 | aspiers | and of course I'm not here | |
| 19:50:38 | efried | okay, nbd | |
| 19:50:46 | aspiers | I wouldn't be so crazy as to be working at 9pm | |
| 19:51:00 | aspiers | he said, while writing functional tests :-/ | |
| 19:53:26 | sean-k-mooney | melwitt: you know one thing people could try is setting the ideal time for the wsgi server to like 300s instead of whatever teh default is | |