Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-05-05
17:42:28 sean-k-mooney although we probly shoudl fix it in oslo too
17:42:30 melwitt ah, true. I was thinking of when I tried literally changing them all + a hacking rule
17:42:40 melwitt +1
17:43:03 sean-k-mooney did you ever raise that with oslo?
17:43:13 sean-k-mooney removing there use of spawn_n
17:43:59 melwitt no. at the time I was so mired in trying to sort the eventlet issue that I didn't think to initiate that
17:44:53 sean-k-mooney ack
17:45:08 sean-k-mooney it looks like spawn_n is pretty common in openstack
17:45:10 sean-k-mooney https://codesearch.opendev.org/?q=spawn_n&i=nope&literal=nope&files=&excludeFiles=&repos=
17:45:31 melwitt I had also intended to put together a more concise repro of the spawn_n problematic behavior to do another eventlet issue but lost steam
17:45:46 melwitt yeah I was thinking that might be the case. I probably checked back then and then forgot
17:47:18 sean-k-mooney so this could be a redherring as to why nothing seams to be resuming the greenthread after the excption is raised when there i s a netowk vif plugged
17:47:26 sean-k-mooney event that fails
17:47:33 mnaser so probably the spawn_n here is causing this somehow ?
17:47:50 sean-k-mooney mnaser: thats what im specualting but not sure
17:48:04 sean-k-mooney im wondiering if we just spawned a fucntion taht raise woudl that trigger this
17:50:26 sean-k-mooney anyway i need to go fo today but it might be worth tryign to repoduce this outside fo nova with a simpel script that use eventles swan_n and raises
17:50:53 sean-k-mooney https://eventlet.net/doc/basic_usage.html#eventlet.spawn_n
17:50:56 mnaser yeah.. I don’t know if I have spare cycles right now for that but I think I should probably file a bug in launchpad at least
17:50:58 sean-k-mooney The same as spawn(), but it’s not possible to know how the function terminated (i.e. no return value or exceptions). This makes execution faster. See spawn_n for more details.
17:51:19 sean-k-mooney https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.spawn_n
17:51:31 melwitt yeah, I'll try to look into this too
17:51:34 sean-k-mooney If an exception is raised in the function, spawn_n prints a stack trace; the print can be disabled by calling eventlet.debug.hub_exceptions() with False.
17:52:17 sean-k-mooney so ya i think this is a high likely hood that this is the probelm
17:52:52 opendevreview Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684
17:53:13 sean-k-mooney if this was c or c++ spwan_n woudl be annotated with [[noreturn]]
17:53:51 sean-k-mooney ok got to go but let me know if this helps
18:27:21 mnaser sean-k-mooney, melwitt: https://bugs.launchpad.net/nova/+bug/1971760
18:28:03 melwitt great, thanks
18:34:51 opendevreview melanie witt proposed openstack/nova stable/wallaby: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840717
18:36:27 opendevreview melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701
18:38:53 opendevreview melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701
18:38:54 opendevreview melanie witt proposed openstack/placement stable/wallaby: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840718
19:23:28 mnaser melwitt, sean-k-mooney: found this https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/compute/manager.py#L479 which goes to https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/network/model.py#L623
19:23:46 mnaser which references https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/network/model.py#L590
19:23:58 mnaser so .spawn_n which calls .spawn
19:28:41 melwitt mnaser: I don't see any spawn_n there, or is that what you're pointing out?
19:30:49 mnaser melwitt: oh right, so _locked_do_build_and_run_instance calls spawn().. and down the line that ends up in wait_for_instance_event() which calls .wait() for the event
19:31:06 mnaser i mean, i am trying to create a reproducer but not really able to get something to fail :(
19:32:00 mnaser https://paste.opendev.org/show/b2msfzi2ASYNwN0Fguqp/
19:32:15 mnaser then kill -USR2 <pid> and no bueno, i dont see those extra threads
19:35:03 melwitt hm ok
19:35:22 mnaser im trying to play with it to reproduce but there's a lot in play i think
19:36:08 melwitt for sure
19:41:20 opendevreview melanie witt proposed openstack/placement stable/yoga: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/placement/+/840728
19:49:48 mnaser i got a full log of a failure
19:50:24 mnaser https://paste.opendev.org/show/bbd1luUAF8B0rwZnl4Cw/
20:32:25 opendevreview melanie witt proposed openstack/placement stable/xena: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/placement/+/840756
21:30:06 opendevreview melanie witt proposed openstack/placement stable/wallaby: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840718
21:30:07 opendevreview melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701
21:59:40 opendevreview melanie witt proposed openstack/nova stable/victoria: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840765
22:04:01 opendevreview melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702
22:04:01 opendevreview melanie witt proposed openstack/placement stable/victoria: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840767
22:07:00 opendevreview melanie witt proposed openstack/nova stable/ussuri: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840771
22:08:57 opendevreview melanie witt proposed openstack/placement stable/victoria: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840767
22:08:58 opendevreview melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702
22:10:59 opendevreview melanie witt proposed openstack/placement stable/ussuri: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840703
22:11:00 opendevreview melanie witt proposed openstack/placement stable/ussuri: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840773
22:24:38 opendevreview melanie witt proposed openstack/nova stable/train: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840777
22:25:47 opendevreview melanie witt proposed openstack/placement stable/train: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840704
22:25:48 opendevreview melanie witt proposed openstack/placement stable/train: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840778
22:29:08 melwitt elodilles: just fyi, tried to backport a placement fix to stable branches, found gates broken, so I rebased them onto fixes for the gate ^ (assuming I didn't mess up). and the gate fixes Depends-On nova stable branch changes that I have also posted
23:13:11 gmann stephenfin: good point on release note of dropping the py3.6/3.7 support in oslo patches, do you think we should have added releasenotes in nova in this thttps://review.opendev.org/c/openstack/nova/+/838943
23:13:20 gmann stephenfin: I can add one
23:39:52 opendevreview Ghanshyam proposed openstack/nova master: Add releasenote about dropping pythin 3.6|7 support https://review.opendev.org/c/openstack/nova/+/840786
23:40:06 gmann stephenfin: ^^
#openstack-nova - 2022-05-06
05:00:30 elodilles_pto melwitt: ack, thanks for the heads up! today i'm on PTO but will look after the patches when I'll have time!
05:47:50 melwitt elodilles_pto: no worry about it, enjoy your PTO :)
05:54:42 opendevreview Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684
06:51:48 opendevreview Balazs Gibizer proposed openstack/nova master: Adapt bindep ubuntu-jammy https://review.opendev.org/c/openstack/nova/+/840822
06:53:53 opendevreview Andrew Bonney proposed openstack/nova stable/yoga: Fix segment-aware scheduling permissions error https://review.opendev.org/c/openstack/nova/+/840732
06:54:36 opendevreview Andrew Bonney proposed openstack/nova stable/xena: Fix eventlet.tpool import https://review.opendev.org/c/openstack/nova/+/840733
07:20:17 gibi clarkb, frickler: now that the py310 job is moved to jammy we need bit different fix for nova to pass the job https://review.opendev.org/c/openstack/nova/+/840822
07:20:30 gibi bauzas, sean-k-mooney, elodilles_pto: ^^
07:21:36 kashyap gibi: What is "jammy"?
07:21:38 kashyap Morning, BTW
07:21:47 kashyap Ah, it's the next Ubuntu release
07:26:50 gibi kashyap: o/
07:26:56 gibi yepp ubuntu 22.04
08:02:54 kashyap gibi: Nice live-debugging on that libvirt bug! :)
08:03:03 kashyap Haha, so many "quirks" we learn: "Use of --live and --persistent together is quirky in certain failure scenarios"
08:03:21 kashyap s/live/async/ :D
08:03:33 gibi kashyap: the doc was helpful about debugging \
08:03:47 gibi and yes I learned a lot just by openning that bug upstream
08:04:12 kashyap Yeah, the debug log filters are great. The `virt-admin` tool is great
08:04:19 kashyap (As it dynamically configures the filters)
08:06:01 gibi yeah that virt-admin tool was totally new to me but seem pretty usefull
08:11:42 kashyap gibi: Don't forget to undo the filters after your debugging; or else your disk can get fill up :)
08:11:50 kashyap To turn off:
08:11:53 kashyap $> virt-admin daemon-log-filters ""
08:11:55 bauzas gibi: made my duty
08:12:27 bauzas fun for a French folk if the new Ubuntu release name is Jammy
08:12:29 gibi kashyap: yeah I will do once the discussion stops in the bug
08:12:45 kashyap Yep
08:12:46 gibi bauzas: it is everybodies duty to make the CI happy
08:12:58 gibi *everybody's
08:13:14 bauzas Jammy and Fred (for Uggla who maybe forgot :p )
08:14:58 bauzas (a French TV documentation series for explaining how the world works :) )
08:15:03 bauzas https://www.youtube.com/watch?v=1vSLW2VKEjc
08:15:37 Uggla bauzas, it was cool ! I know Jammy is still doing such kind of things on Youtube.

Earlier   Later