| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-05 | |||
| 17:43:03 | sean-k-mooney | did you ever raise that with oslo? | |
| 17:43:13 | sean-k-mooney | removing there use of spawn_n | |
| 17:43:59 | melwitt | no. at the time I was so mired in trying to sort the eventlet issue that I didn't think to initiate that | |
| 17:44:53 | sean-k-mooney | ack | |
| 17:45:08 | sean-k-mooney | it looks like spawn_n is pretty common in openstack | |
| 17:45:10 | sean-k-mooney | https://codesearch.opendev.org/?q=spawn_n&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 17:45:31 | melwitt | I had also intended to put together a more concise repro of the spawn_n problematic behavior to do another eventlet issue but lost steam | |
| 17:45:46 | melwitt | yeah I was thinking that might be the case. I probably checked back then and then forgot | |
| 17:47:18 | sean-k-mooney | so this could be a redherring as to why nothing seams to be resuming the greenthread after the excption is raised when there i s a netowk vif plugged | |
| 17:47:26 | sean-k-mooney | event that fails | |
| 17:47:33 | mnaser | so probably the spawn_n here is causing this somehow ? | |
| 17:47:50 | sean-k-mooney | mnaser: thats what im specualting but not sure | |
| 17:48:04 | sean-k-mooney | im wondiering if we just spawned a fucntion taht raise woudl that trigger this | |
| 17:50:26 | sean-k-mooney | anyway i need to go fo today but it might be worth tryign to repoduce this outside fo nova with a simpel script that use eventles swan_n and raises | |
| 17:50:53 | sean-k-mooney | https://eventlet.net/doc/basic_usage.html#eventlet.spawn_n | |
| 17:50:56 | mnaser | yeah.. I don’t know if I have spare cycles right now for that but I think I should probably file a bug in launchpad at least | |
| 17:50:58 | sean-k-mooney | The same as spawn(), but it’s not possible to know how the function terminated (i.e. no return value or exceptions). This makes execution faster. See spawn_n for more details. | |
| 17:51:19 | sean-k-mooney | https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.spawn_n | |
| 17:51:31 | melwitt | yeah, I'll try to look into this too | |
| 17:51:34 | sean-k-mooney | If an exception is raised in the function, spawn_n prints a stack trace; the print can be disabled by calling eventlet.debug.hub_exceptions() with False. | |
| 17:52:17 | sean-k-mooney | so ya i think this is a high likely hood that this is the probelm | |
| 17:52:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684 | |
| 17:53:13 | sean-k-mooney | if this was c or c++ spwan_n woudl be annotated with [[noreturn]] | |
| 17:53:51 | sean-k-mooney | ok got to go but let me know if this helps | |
| 18:27:21 | mnaser | sean-k-mooney, melwitt: https://bugs.launchpad.net/nova/+bug/1971760 | |
| 18:28:03 | melwitt | great, thanks | |
| 18:34:51 | opendevreview | melanie witt proposed openstack/nova stable/wallaby: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840717 | |
| 18:36:27 | opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | |
| 18:38:53 | opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | |
| 18:38:54 | opendevreview | melanie witt proposed openstack/placement stable/wallaby: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840718 | |
| 19:23:28 | mnaser | melwitt, sean-k-mooney: found this https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/compute/manager.py#L479 which goes to https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/network/model.py#L623 | |
| 19:23:46 | mnaser | which references https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/network/model.py#L590 | |
| 19:23:58 | mnaser | so .spawn_n which calls .spawn | |
| 19:28:41 | melwitt | mnaser: I don't see any spawn_n there, or is that what you're pointing out? | |
| 19:30:49 | mnaser | melwitt: oh right, so _locked_do_build_and_run_instance calls spawn().. and down the line that ends up in wait_for_instance_event() which calls .wait() for the event | |
| 19:31:06 | mnaser | i mean, i am trying to create a reproducer but not really able to get something to fail :( | |
| 19:32:00 | mnaser | https://paste.opendev.org/show/b2msfzi2ASYNwN0Fguqp/ | |
| 19:32:15 | mnaser | then kill -USR2 <pid> and no bueno, i dont see those extra threads | |
| 19:35:03 | melwitt | hm ok | |
| 19:35:22 | mnaser | im trying to play with it to reproduce but there's a lot in play i think | |
| 19:36:08 | melwitt | for sure | |
| 19:41:20 | opendevreview | melanie witt proposed openstack/placement stable/yoga: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/placement/+/840728 | |
| 19:49:48 | mnaser | i got a full log of a failure | |
| 19:50:24 | mnaser | https://paste.opendev.org/show/bbd1luUAF8B0rwZnl4Cw/ | |
| 20:32:25 | opendevreview | melanie witt proposed openstack/placement stable/xena: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/placement/+/840756 | |
| 21:30:06 | opendevreview | melanie witt proposed openstack/placement stable/wallaby: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840718 | |
| 21:30:07 | opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | |
| 21:59:40 | opendevreview | melanie witt proposed openstack/nova stable/victoria: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840765 | |
| 22:04:01 | opendevreview | melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702 | |
| 22:04:01 | opendevreview | melanie witt proposed openstack/placement stable/victoria: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840767 | |
| 22:07:00 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840771 | |
| 22:08:57 | opendevreview | melanie witt proposed openstack/placement stable/victoria: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840767 | |
| 22:08:58 | opendevreview | melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702 | |
| 22:10:59 | opendevreview | melanie witt proposed openstack/placement stable/ussuri: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840703 | |
| 22:11:00 | opendevreview | melanie witt proposed openstack/placement stable/ussuri: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840773 | |
| 22:24:38 | opendevreview | melanie witt proposed openstack/nova stable/train: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840777 | |
| 22:25:47 | opendevreview | melanie witt proposed openstack/placement stable/train: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840704 | |
| 22:25:48 | opendevreview | melanie witt proposed openstack/placement stable/train: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840778 | |
| 22:29:08 | melwitt | elodilles: just fyi, tried to backport a placement fix to stable branches, found gates broken, so I rebased them onto fixes for the gate ^ (assuming I didn't mess up). and the gate fixes Depends-On nova stable branch changes that I have also posted | |
| 23:13:11 | gmann | stephenfin: good point on release note of dropping the py3.6/3.7 support in oslo patches, do you think we should have added releasenotes in nova in this thttps://review.opendev.org/c/openstack/nova/+/838943 | |
| 23:13:20 | gmann | stephenfin: I can add one | |
| 23:39:52 | opendevreview | Ghanshyam proposed openstack/nova master: Add releasenote about dropping pythin 3.6|7 support https://review.opendev.org/c/openstack/nova/+/840786 | |
| 23:40:06 | gmann | stephenfin: ^^ | |
| #openstack-nova - 2022-05-06 | |||
| 05:00:30 | elodilles_pto | melwitt: ack, thanks for the heads up! today i'm on PTO but will look after the patches when I'll have time! | |
| 05:47:50 | melwitt | elodilles_pto: no worry about it, enjoy your PTO :) | |
| 05:54:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684 | |
| 06:51:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt bindep ubuntu-jammy https://review.opendev.org/c/openstack/nova/+/840822 | |
| 06:53:53 | opendevreview | Andrew Bonney proposed openstack/nova stable/yoga: Fix segment-aware scheduling permissions error https://review.opendev.org/c/openstack/nova/+/840732 | |
| 06:54:36 | opendevreview | Andrew Bonney proposed openstack/nova stable/xena: Fix eventlet.tpool import https://review.opendev.org/c/openstack/nova/+/840733 | |
| 07:20:17 | gibi | clarkb, frickler: now that the py310 job is moved to jammy we need bit different fix for nova to pass the job https://review.opendev.org/c/openstack/nova/+/840822 | |
| 07:20:30 | gibi | bauzas, sean-k-mooney, elodilles_pto: ^^ | |
| 07:21:36 | kashyap | gibi: What is "jammy"? | |
| 07:21:38 | kashyap | Morning, BTW | |
| 07:21:47 | kashyap | Ah, it's the next Ubuntu release | |
| 07:26:50 | gibi | kashyap: o/ | |
| 07:26:56 | gibi | yepp ubuntu 22.04 | |
| 08:02:54 | kashyap | gibi: Nice live-debugging on that libvirt bug! :) | |
| 08:03:03 | kashyap | Haha, so many "quirks" we learn: "Use of --live and --persistent together is quirky in certain failure scenarios" | |
| 08:03:21 | kashyap | s/live/async/ :D | |
| 08:03:33 | gibi | kashyap: the doc was helpful about debugging \ | |
| 08:03:47 | gibi | and yes I learned a lot just by openning that bug upstream | |
| 08:04:12 | kashyap | Yeah, the debug log filters are great. The `virt-admin` tool is great | |
| 08:04:19 | kashyap | (As it dynamically configures the filters) | |
| 08:06:01 | gibi | yeah that virt-admin tool was totally new to me but seem pretty usefull | |
| 08:11:42 | kashyap | gibi: Don't forget to undo the filters after your debugging; or else your disk can get fill up :) | |
| 08:11:50 | kashyap | To turn off: | |
| 08:11:53 | kashyap | $> virt-admin daemon-log-filters "" | |
| 08:11:55 | bauzas | gibi: made my duty | |
| 08:12:27 | bauzas | fun for a French folk if the new Ubuntu release name is Jammy | |
| 08:12:29 | gibi | kashyap: yeah I will do once the discussion stops in the bug | |
| 08:12:45 | kashyap | Yep | |
| 08:12:46 | gibi | bauzas: it is everybodies duty to make the CI happy | |
| 08:12:58 | gibi | *everybody's | |
| 08:13:14 | bauzas | Jammy and Fred (for Uggla who maybe forgot :p ) | |
| 08:14:58 | bauzas | (a French TV documentation series for explaining how the world works :) ) | |
| 08:15:03 | bauzas | https://www.youtube.com/watch?v=1vSLW2VKEjc | |
| 08:15:37 | Uggla | bauzas, it was cool ! I know Jammy is still doing such kind of things on Youtube. | |
| 08:15:57 | bauzas | actually, he's named Jamy with a single 'm' :p | |
| 08:16:07 | bauzas | and yeah I know | |
| 08:16:35 | Uggla | bauzas, I liked the truck ! | |