| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-28 | |||
| 11:46:03 | sean-k-mooney | right please dont | |
| 11:46:06 | ralonsoh | however that doesn't address the problem of "concurrent port binding" | |
| 11:46:10 | sean-k-mooney | at least not with a periodic | |
| 11:46:11 | ralonsoh | don't what? | |
| 11:46:18 | sean-k-mooney | try an fix it form neutron | |
| 11:46:45 | ralonsoh | ok but the problem is anytime we have this issue, the bug is bug Neutron | |
| 11:46:50 | sean-k-mooney | we need to fix this form the nova side to avoid posible races between nova and neturon | |
| 11:47:20 | sean-k-mooney | right so there are two ways to fix this in nova. 1 make sure we delete the inactive port bidning when we revert | |
| 11:47:31 | sean-k-mooney | i think we try that already today but it can fial | |
| 11:47:41 | sean-k-mooney | second when we live migrate or try to create a portbidning | |
| 11:47:49 | sean-k-mooney | and it already exits delete and recreate | |
| 11:48:03 | sean-k-mooney | that will prevent this form breaking in the future | |
| 11:48:09 | frickler | the failure in osc isn't related to migration. it happens on initial server create. so I don't understand what could cause the duplicate there | |
| 11:48:37 | sean-k-mooney | if the initall port create faild and we resuchlde then we will try binding it to a second host | |
| 11:48:42 | opendevreview | Sergii Golovatiuk proposed openstack/nova master: Replace "db archive" with "db archive_deleted_raws" https://review.opendev.org/c/openstack/nova/+/847963 | |
| 11:48:48 | sean-k-mooney | we wont create a second binding | |
| 11:48:54 | sean-k-mooney | we will just update the host-id | |
| 11:49:20 | sean-k-mooney | but if neutron is still binding it form the first failed attempet we woudl get a concurrent error | |
| 11:49:39 | sean-k-mooney | frickler: did you check the logs to see if the vm was retired on a second host? | |
| 11:51:55 | frickler | sean-k-mooney: it is a single-node job, I would be surprised if that happened | |
| 11:53:38 | sean-k-mooney | frickler: ok well the only other thing i can think of is the client retry | |
| 11:54:03 | sean-k-mooney | ralonsoh: could this happen if nova retryed creating the port binding because thet inital call timed out | |
| 11:54:08 | sean-k-mooney | i think we try 3 times | |
| 11:54:54 | sean-k-mooney | this being "Concurrent port binding operations failed on port ..." | |
| 12:19:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add regression test for bug 1838392 https://review.opendev.org/c/openstack/nova/+/847965 | |
| 13:08:13 | bauzas | gibi: sean-k-mooney: jkulik: fwiw, this is not a new meat https://review.opendev.org/c/openstack/nova-specs/+/130005 | |
| 13:08:36 | sean-k-mooney | bauzas: yep i know | |
| 13:08:50 | bauzas | sure, but I provided the existing context | |
| 13:09:02 | sean-k-mooney | there are other specs more recnet then that | |
| 13:09:05 | bauzas | just look at the gerrit comments | |
| 13:09:30 | sean-k-mooney | but ya we have rejected dynmic server groups and adding other affinity policies in the past | |
| 13:10:14 | bauzas | yes and no, this is just the fact that the spec was split https://review.opendev.org/c/openstack/nova-specs/+/139272 and https://review.opendev.org/c/openstack/nova-specs/+/136487 | |
| 13:10:22 | bauzas | look again about the comments | |
| 13:10:44 | bauzas | sean-k-mooney: not for you, but rather jkulik | |
| 13:11:15 | jkulik | bauzas: thanks, will take a look | |
| 13:11:20 | sean-k-mooney | bauzas: ack | |
| 13:13:22 | bauzas | jkulik: there was by then a try to have a new API instead of server groups https://review.opendev.org/c/openstack/nova-specs/+/183837/4/specs/liberty/approved/generic-scheduling-policies.rst | |
| 13:13:54 | sean-k-mooney | bauzas: im not sure how that would help | |
| 13:14:07 | sean-k-mooney | the issue with server groups is adding an instance that violates the policy | |
| 13:14:10 | bauzas | just saying this is a can of worms | |
| 13:14:30 | sean-k-mooney | it is | |
| 13:14:38 | bauzas | sean-k-mooney: the problem with server groups is that if we touch it, it creates more races than the ones it fixes | |
| 13:14:42 | sean-k-mooney | but that is a sperate topic form the orginal thing that jkulik raised | |
| 13:15:00 | bauzas | sean-k-mooney: well, this is all about colocality | |
| 13:15:12 | bauzas | we express this in Nova with server groups | |
| 13:15:23 | sean-k-mooney | well affinity and anti affity | |
| 13:15:27 | sean-k-mooney | with differnt granularity | |
| 13:15:29 | sean-k-mooney | but yes | |
| 13:15:52 | bauzas | but the question remains about the best UX we may have | |
| 13:18:24 | bauzas | anyway, me goes back at bug scrub | |
| 14:29:45 | Uggla | question, from the api, I try to check that a scheduling is impossible. To do that I look at 'No valid host found for unshelve instance' in the log. My test seems to work, but I have to wait before checking the logs. Is there a proper way to do that ? | |
| 14:30:54 | Uggla | bauzas, gibi ^ | |
| 14:33:03 | sean-k-mooney | using notifications | |
| 14:33:15 | sean-k-mooney | but in general we dont tend to use logs in tests | |
| 14:33:25 | sean-k-mooney | we do sometimes but there are often better way to do that | |
| 14:33:45 | Uggla | is _wait_for_action_fail_completion a possible option ? | |
| 14:43:15 | gibi | Uggla: you wait for the server to go to ERROR state then you can check the fault in the server to see if it is a no valid host | |
| 14:43:22 | gibi | give me a sec and I will find an example | |
| 14:45:23 | gibi | Uggla: for example https://github.com/openstack/nova/blob/c53ec4e48884235566962bc934cbf292ad5b67b8/nova/tests/functional/test_servers.py#L4100-L4108 | |
| 14:45:28 | Uggla | gibi, ok probably better that what I have just done using wait_for_assert(). | |
| 15:07:24 | bauzas | Uggla: not sure I understand your question | |
| 15:07:36 | bauzas | you're asking how nova-api could know there is a scheduling error ? | |
| 15:07:53 | Uggla | bauzas, no worries Gibi is helping me right now. | |
| 15:08:06 | bauzas | oh, for testing | |
| 15:08:41 | sean-k-mooney | yep testing | |
| 15:08:46 | sean-k-mooney | presumable functional testing | |
| 15:08:51 | sean-k-mooney | rather then unit | |
| 15:11:24 | gibi | yepp func testing | |
| 15:11:58 | gibi | Uggla just found the limitation of the our nova.tests.functional.integrated_helpers.InstanceHelperMixin._wait_for_instance_action_event assert | |
| 15:12:18 | gibi | it always checks the first action of a given type from the list of instance actions | |
| 15:12:27 | gibi | and he had two unshelve actions in the test case | |
| 15:12:35 | gibi | and the assert only checked the first | |
| 15:12:43 | gibi | even though the second was in error state | |
| 15:29:27 | bauzas | reminder: nova meeting in 30 mins | |
| 15:31:07 | sean-k-mooney | oh fun | |
| 15:31:16 | sean-k-mooney | i was not aware we had that limitation | |
| 15:40:06 | gibi | me neither, but now Uggla can improve on that :) | |
| 15:42:15 | Uggla | gibi, sean-k-mooney , currently not completely sure but no the code looks ok. It is more the event I'm looking for which is not the good one. | |
| 15:43:38 | gibi | it can be that the listing of the instance actions are not stable so sometimes the code founds the proper unshelve action sometimes not | |
| 15:48:34 | gibi | I mean the sorting is not stable | |
| 16:00:11 | bauzas | #startmeeting nova | |
| 16:00:11 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:11 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:11 | opendevmeet | Meeting started Tue Jun 28 16:00:11 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:15 | bauzas | hello everyone | |
| 16:01:09 | bauzas | mmmm | |
| 16:01:21 | bauzas | heyhoooooooo | |
| 16:02:00 | gibi | o/ | |
| 16:02:19 | bauzas | hah, I hear some sound | |
| 16:02:30 | gibi | crickets ? | |
| 16:02:42 | bauzas | I'm maybe in a cave | |
| 16:03:22 | bauzas | gibi: not sure we have quorum for today's meeting :( | |
| 16:03:23 | elodilles | o/ | |
| 16:03:37 | gibi | sean-k-mooney already gone for today | |
| 16:03:58 | gibi | I guess melwitt is still on PTO | |
| 16:03:59 | bauzas | hah | |
| 16:04:05 | bauzas | yes indeed + artom | |
| 16:04:30 | bauzas | what would you want ? | |
| 16:05:03 | gibi | do we have somebody here for the todays nova meeting to talk about specific things? (if not then we can close this) | |
| 16:05:39 | bauzas | we still have one critical bug | |