Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-29
17:15:04 aspiers +1
17:15:50 kashyap aspiers: Ah, thanks. My focus was in Gerrit, and I saw the "Update" pop-up (your comment). I was expecting a firm -1 :D
17:16:02 aspiers Oh ye of little faith :)
17:16:03 kashyap Okay, now I cook food
17:16:07 aspiers Good plan
17:35:27 ganso lyarwood, melwitt, dansmith: Hello! When you have a minute, could you please take a look at those backports that are missing just a final +2? https://review.opendev.org/#/c/652153 , https://review.opendev.org/#/c/658136 and https://review.opendev.org/#/c/661349 ? Thanks in advance! =)
17:35:30 ganso tonyb: thanks a lot =D
17:49:23 openstackgerrit melanie witt proposed openstack/nova master: Follow up for counting quota usage from placement https://review.opendev.org/662056
18:24:03 melwitt mriedem: replied to your comments on https://review.opendev.org/#/c/638073
18:24:22 melwitt and I dunno if you saw, but it looks like https://review.opendev.org/661391 fixes the novnc tests on the multi-cell job
18:36:34 openstackgerrit Artom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881
18:36:34 openstackgerrit Artom Lifshitz proposed openstack/nova master: Run revert resize tests in nova-live-migration https://review.opendev.org/653498
18:36:35 openstackgerrit Artom Lifshitz proposed openstack/nova master: [DNM] use iptables in nova-multinode https://review.opendev.org/660782
18:37:07 artom mriedem, ^^ when you get some time, assuming unit tests pass (I've fixed all of them, normally). I think sean-k-mooney and I have finally figured it out.
18:37:49 artom The DNM patch is... optional? I'm not sure what we want to do upstream, but it's good to have it on top to show that both hybrid and normal plug work
18:47:53 sean-k-mooney the DNM patch change the job to use iptables. if we wanted to merge it we should add a second job that use iptable rather then changing multinode job to use it instead of the defualt contrack driver
18:49:11 konetzed will the scheduler honor setting ram_weight_multiplier as extra metadata on a host aggregate? The goal is just to have one host aggregate stack vs spread.
18:50:52 sean-k-mooney konetzed: i dont think so https://github.com/openstack/nova/blob/master/nova/scheduler/weights/ram.py
18:51:33 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L1089
18:51:43 melwitt konetzed: to do that, I think you'd have to set ram_weight_multiplier as desired in the nova.conf of the compute hosts that are a member of the host aggregate you want to stack vs spread
18:51:47 sean-k-mooney actully look like it might
18:52:11 melwitt ok, I didn't know that :\
18:52:45 sean-k-mooney neither did i i assume we broke that when we broke the allocation ratios
18:52:49 sean-k-mooney i guess not
18:53:42 sean-k-mooney its proably suppor fragile if the host is in two aggrates with different values
18:54:57 mriedem you can have per-aggregate weights
18:55:27 sean-k-mooney well from the code yes but if a host is in two aggrates and they have differnt values im not sure what we do
18:55:43 konetzed melwitt: sean-k-mooney: thanks for the quick answer.
18:55:54 mriedem https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights
18:56:00 mriedem "If more than one value is found for a host in aggregate metadata, the minimum value will be used."
18:56:02 mriedem depends on the weigher
18:56:05 mriedem but the docs are all tehre
18:56:06 mriedem *there
18:56:10 mriedem per-aggregate weights is new in stein
18:56:13 mriedem yikun added it
18:56:27 melwitt konetzed ^ updated answer :P sorry
18:57:12 konetzed mriedem: thanks. Looks like i need to update from Rocky ;)
19:26:34 mriedem melwitt: ack i'll get back to it in a bit
19:26:42 melwitt thanks
19:27:01 mriedem unrelated plug for a question about the first stable stein release for anyone that has input http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006617.html
19:27:11 mriedem otherwise i'll probably just punt and un-WIP the release patch
19:27:52 sean-k-mooney mriedem: oh ya i ment to reply. i think adding a known issuse release not makes sense
19:28:01 sean-k-mooney im just not sure what it shoudl say
19:33:16 mriedem btw is this the heartbeat thing we'd expect to see in the n-api logs for that bug http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/screen-n-api.txt.gz?level=TRACE#_May_29_18_00_29_897709 ?
19:33:25 mriedem because it's all over these tempest-pg-full logs
19:36:15 melwitt I haven't actually seen that log for this.. when I reproduced it in devstack I only saw the "connection reset" error message from oslo.messaging. and some other messages about retrying/reconnecting
19:37:35 mriedem i see lots of "missed heartbeats from client, timeout: 60s" in the rabbit logs
19:37:38 mriedem http://logs.openstack.org/28/662028/1/experimental/tempest-pg-full/91f6094/controller/logs/rabbitmq/rabbit@ubuntu-bionic-rax-iad-0006739854_log.txt.gz
19:37:42 sean-k-mooney it looks like the tcp connection is also being closed
19:38:11 sean-k-mooney mriedem: ya that is expected
19:39:22 sean-k-mooney if we dont disabel monkey patching or disabel the heartbeat then we will have missed heartbeat message in rabbitmq
19:40:41 mriedem i guess my point is there was concern about not hitting this in the gate
19:40:55 mriedem even though we run with uwsgi in devstack and eventlet monkey-patched
19:41:38 sean-k-mooney well in the gate we should see this since we have 1 thread per wsgi process
19:41:48 sean-k-mooney but the api sould actuly still work
19:42:12 melwitt yeah, the gate is normally too busy with activity to cause the heartbeat interruption where the wsgi app pauses bc of lack of activity
19:43:20 melwitt so... I dunno how that's showing up in tempest-pg-full :\
19:43:40 zigo mriedem: Hey matt, thanks again for your patch, it fixed it for us. I've uploaded the fix to Sid, and hopefully it will be part of Buster (currently filling-up an unblock bug to the Debian release team).
19:44:35 mriedem zigo: yw. as noted on the review, i'll likely be modifying that patch a bit.
19:44:37 sean-k-mooney melwitt: wll its not printing constantly so it posible that ocationally we do time out even in the gate
19:45:17 melwitt o
19:48:52 mriedem brb
19:50:14 efried aspiers, kashyap: Minor tweaks to https://review.opendev.org/#/c/655193/ if either of you is around.
19:50:24 aspiers kashyap's gone I think
19:50:31 aspiers and of course I'm not here
19:50:38 efried okay, nbd
19:50:46 aspiers I wouldn't be so crazy as to be working at 9pm
19:51:00 aspiers he said, while writing functional tests :-/
19:53:26 sean-k-mooney melwitt: you know one thing people could try is setting the ideal time for the wsgi server to like 300s instead of whatever teh default is
19:54:11 sean-k-mooney if you are not reciving at least 1 api request every 5 minutes to keep it alive you proabley dont need the performace boost from moneypatching
19:55:09 sean-k-mooney you will endup useing more ram as the wsgi app will be loaded more often but its tradoffs nomater what we do
19:55:11 melwitt well... what about in the middle of the night, when no one's using the api potentially? I feels like any deployment, no matter how large or busy, will have some down time at some point during the day
19:56:25 sean-k-mooney in that case it will suspend after 5 mins and then the next request it recived it will reconnect
19:56:57 melwitt ok, so you're still talking in the context of threads=1 and just avoiding the scary heartbeat messages more
19:57:11 sean-k-mooney but if people want to reduce how often that log message is printed they can tweak how agress the wsgi server is at sleeping
19:57:17 sean-k-mooney ya
19:57:25 melwitt I thought you were suggesting increasing the idle timeout as a way of allowing threads > 1 to work
19:57:27 melwitt ok
19:57:35 aspiers efried: I finally found MAX_INT in placement.db.constants, but that's not available from the unit testing testenvs
19:57:55 efried aspiers: Swear there was somewhere else we were using that.
19:57:56 aspiers using nova.db.constants for now
19:58:01 aspiers but that's not really correct
19:58:06 aspiers even if they happen to be the same value
19:58:41 aspiers efried: it's available to functional testenvs, but I need it in one unit test also
19:58:51 melwitt mriedem: I can take a stab at the stein reno and then have sean-k-mooney correct me because sean knows more than I do about it at this point
19:58:52 efried aspiers: You need it in your code too
19:59:10 sean-k-mooney aspiers: you could just use 2**64 or 2**32
19:59:27 aspiers efried: well sure, but presumably it's safe to use placement from nova code?
19:59:34 efried not at all
19:59:41 aspiers huh
19:59:58 sean-k-mooney aspiers: we cant import random parts of placement in nova
20:00:04 aspiers OK
20:00:14 efried Right. Placement, from nova's perspective, is supposed to be purely a REST interface. Absolutely nothing else.
20:00:25 efried I think we're importing a fixture to make some testing easier
20:00:36 efried but prod code should be purely REST touchpoints.
20:00:38 aspiers Seems totally reasonable to me if it exported constants at least, if nothing else
20:00:48 efried yeah, I could get behind that in theory.
20:00:56 efried There has been talk of making a placement
20:01:00 efried placement-lib project
20:01:01 sean-k-mooney why not just creat the constant in nova

Earlier   Later