| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-28 | |||
| 10:33:17 | jkulik | fyi, this is how we built it downstream https://github.com/sapcc/nova/commit/7220be3968ee1dd257c9add88228cc5bb9857795 (+ some commits afterwards to fix certain stuff) | |
| 10:41:19 | sean-k-mooney | i see that still has the same problem | |
| 10:41:42 | frickler | does that ring a bell for someone? "nova.exception.InternalError: Unexpected vif_type=unbound" unstable failure in OSC jobs, shows as JSONDecodeError https://zuul.opendev.org/t/openstack/build/181d8177eab5428a82facc4d95ce00e2 | |
| 10:42:13 | sean-k-mooney | vif_type unbound is what the netron itnerface has before you set the host-id | |
| 10:43:05 | sean-k-mooney | frickler: so } openstackclient.tests.functional.compute.v2.test_server.ServerTests.test_server_attach_detach_floating_ip might be racing | |
| 10:43:09 | sean-k-mooney | with the server boot | |
| 10:43:29 | sean-k-mooney | if it has not finished booting when you try to attach the floating ip then you woudl get that issue i guess | |
| 10:43:56 | frickler | oh, so yet another set of tests needing wait-for-ssh things | |
| 10:44:11 | sean-k-mooney | maybe have not looked at the test yet | |
| 10:44:34 | sean-k-mooney | its not nessiarly sshable | |
| 10:44:38 | sean-k-mooney | it woudl need to be active | |
| 10:45:01 | sean-k-mooney | so this is the test https://github.com/openstack/python-openstackclient/blob/master/openstackclient/tests/functional/compute/v2/test_server.py#L339= | |
| 10:45:16 | sean-k-mooney | it shoud be waiting for active | |
| 10:45:39 | sean-k-mooney | it looks like the error is coming form self.server_create | |
| 10:46:12 | sean-k-mooney | ah from here https://github.com/openstack/python-openstackclient/blob/20e7b01af8f0fb4cf0f4af253270ad470926ba4e/openstackclient/tests/functional/compute/v2/common.py#L89= | |
| 10:46:46 | sean-k-mooney | frickler: so its assuming that a value will be populated in the output i guess | |
| 10:47:28 | sean-k-mooney | from the trace it looks like it got an empty responce of something like that | |
| 10:47:54 | sean-k-mooney | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) | |
| 10:48:42 | sean-k-mooney | frickler: without a request id it might not be easy to see what the api returned | |
| 10:49:39 | sean-k-mooney | 136.246758s that seamed to take a very long time | |
| 10:49:49 | sean-k-mooney | is there a timeout for wait? | |
| 11:15:58 | frickler | neutron says "Concurrent port binding operations failed on port 5b1348e9-4048-4b39-a1df-1161a798052e" before nova fails, so more likely an issue on their side | |
| 11:19:08 | sean-k-mooney | concurrent? | |
| 11:19:13 | sean-k-mooney | how | |
| 11:19:35 | sean-k-mooney | that could only happne if we failed reschulded and tried to bind again | |
| 11:19:47 | sean-k-mooney | but they were sitll binding the port to the first host | |
| 11:20:08 | sean-k-mooney | so i guess yes that woudl be a neutorn issue | |
| 11:20:28 | sean-k-mooney | that is not as far as i am aware one of the exctpiotn they are allowed to raise at teh api level | |
| 11:42:33 | frickler | that's just a warning in the q-svc log and iiuc they retry and succeed after that. but it may be that they send a notification to nova about the first attempt anyway | |
| 11:43:55 | sean-k-mooney | maybe i know ralonsoh has a wip patch for somethign else that i dont think will actully help but a race was mentioned in cotext to that | |
| 11:44:58 | sean-k-mooney | frickler: https://review.opendev.org/c/openstack/neutron/+/846422/3 | |
| 11:45:09 | sean-k-mooney | https://bugs.launchpad.net/neutron/+bug/1979072 | |
| 11:45:17 | sean-k-mooney | that shoudl actully be fixed in nova | |
| 11:45:30 | ralonsoh | sean-k-mooney, right, this is just a WIP patch | |
| 11:45:51 | ralonsoh | trying to address an issue that is on the Neutron side | |
| 11:46:03 | sean-k-mooney | right please dont | |
| 11:46:06 | ralonsoh | however that doesn't address the problem of "concurrent port binding" | |
| 11:46:10 | sean-k-mooney | at least not with a periodic | |
| 11:46:11 | ralonsoh | don't what? | |
| 11:46:18 | sean-k-mooney | try an fix it form neutron | |
| 11:46:45 | ralonsoh | ok but the problem is anytime we have this issue, the bug is bug Neutron | |
| 11:46:50 | sean-k-mooney | we need to fix this form the nova side to avoid posible races between nova and neturon | |
| 11:47:20 | sean-k-mooney | right so there are two ways to fix this in nova. 1 make sure we delete the inactive port bidning when we revert | |
| 11:47:31 | sean-k-mooney | i think we try that already today but it can fial | |
| 11:47:41 | sean-k-mooney | second when we live migrate or try to create a portbidning | |
| 11:47:49 | sean-k-mooney | and it already exits delete and recreate | |
| 11:48:03 | sean-k-mooney | that will prevent this form breaking in the future | |
| 11:48:09 | frickler | the failure in osc isn't related to migration. it happens on initial server create. so I don't understand what could cause the duplicate there | |
| 11:48:37 | sean-k-mooney | if the initall port create faild and we resuchlde then we will try binding it to a second host | |
| 11:48:42 | opendevreview | Sergii Golovatiuk proposed openstack/nova master: Replace "db archive" with "db archive_deleted_raws" https://review.opendev.org/c/openstack/nova/+/847963 | |
| 11:48:48 | sean-k-mooney | we wont create a second binding | |
| 11:48:54 | sean-k-mooney | we will just update the host-id | |
| 11:49:20 | sean-k-mooney | but if neutron is still binding it form the first failed attempet we woudl get a concurrent error | |
| 11:49:39 | sean-k-mooney | frickler: did you check the logs to see if the vm was retired on a second host? | |
| 11:51:55 | frickler | sean-k-mooney: it is a single-node job, I would be surprised if that happened | |
| 11:53:38 | sean-k-mooney | frickler: ok well the only other thing i can think of is the client retry | |
| 11:54:03 | sean-k-mooney | ralonsoh: could this happen if nova retryed creating the port binding because thet inital call timed out | |
| 11:54:08 | sean-k-mooney | i think we try 3 times | |
| 11:54:54 | sean-k-mooney | this being "Concurrent port binding operations failed on port ..." | |
| 12:19:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add regression test for bug 1838392 https://review.opendev.org/c/openstack/nova/+/847965 | |
| 13:08:13 | bauzas | gibi: sean-k-mooney: jkulik: fwiw, this is not a new meat https://review.opendev.org/c/openstack/nova-specs/+/130005 | |
| 13:08:36 | sean-k-mooney | bauzas: yep i know | |
| 13:08:50 | bauzas | sure, but I provided the existing context | |
| 13:09:02 | sean-k-mooney | there are other specs more recnet then that | |
| 13:09:05 | bauzas | just look at the gerrit comments | |
| 13:09:30 | sean-k-mooney | but ya we have rejected dynmic server groups and adding other affinity policies in the past | |
| 13:10:14 | bauzas | yes and no, this is just the fact that the spec was split https://review.opendev.org/c/openstack/nova-specs/+/139272 and https://review.opendev.org/c/openstack/nova-specs/+/136487 | |
| 13:10:22 | bauzas | look again about the comments | |
| 13:10:44 | bauzas | sean-k-mooney: not for you, but rather jkulik | |
| 13:11:15 | jkulik | bauzas: thanks, will take a look | |
| 13:11:20 | sean-k-mooney | bauzas: ack | |
| 13:13:22 | bauzas | jkulik: there was by then a try to have a new API instead of server groups https://review.opendev.org/c/openstack/nova-specs/+/183837/4/specs/liberty/approved/generic-scheduling-policies.rst | |
| 13:13:54 | sean-k-mooney | bauzas: im not sure how that would help | |
| 13:14:07 | sean-k-mooney | the issue with server groups is adding an instance that violates the policy | |
| 13:14:10 | bauzas | just saying this is a can of worms | |
| 13:14:30 | sean-k-mooney | it is | |
| 13:14:38 | bauzas | sean-k-mooney: the problem with server groups is that if we touch it, it creates more races than the ones it fixes | |
| 13:14:42 | sean-k-mooney | but that is a sperate topic form the orginal thing that jkulik raised | |
| 13:15:00 | bauzas | sean-k-mooney: well, this is all about colocality | |
| 13:15:12 | bauzas | we express this in Nova with server groups | |
| 13:15:23 | sean-k-mooney | well affinity and anti affity | |
| 13:15:27 | sean-k-mooney | with differnt granularity | |
| 13:15:29 | sean-k-mooney | but yes | |
| 13:15:52 | bauzas | but the question remains about the best UX we may have | |
| 13:18:24 | bauzas | anyway, me goes back at bug scrub | |
| 14:29:45 | Uggla | question, from the api, I try to check that a scheduling is impossible. To do that I look at 'No valid host found for unshelve instance' in the log. My test seems to work, but I have to wait before checking the logs. Is there a proper way to do that ? | |
| 14:30:54 | Uggla | bauzas, gibi ^ | |
| 14:33:03 | sean-k-mooney | using notifications | |
| 14:33:15 | sean-k-mooney | but in general we dont tend to use logs in tests | |
| 14:33:25 | sean-k-mooney | we do sometimes but there are often better way to do that | |
| 14:33:45 | Uggla | is _wait_for_action_fail_completion a possible option ? | |
| 14:43:15 | gibi | Uggla: you wait for the server to go to ERROR state then you can check the fault in the server to see if it is a no valid host | |
| 14:43:22 | gibi | give me a sec and I will find an example | |
| 14:45:23 | gibi | Uggla: for example https://github.com/openstack/nova/blob/c53ec4e48884235566962bc934cbf292ad5b67b8/nova/tests/functional/test_servers.py#L4100-L4108 | |
| 14:45:28 | Uggla | gibi, ok probably better that what I have just done using wait_for_assert(). | |
| 15:07:24 | bauzas | Uggla: not sure I understand your question | |
| 15:07:36 | bauzas | you're asking how nova-api could know there is a scheduling error ? | |
| 15:07:53 | Uggla | bauzas, no worries Gibi is helping me right now. | |
| 15:08:06 | bauzas | oh, for testing | |
| 15:08:41 | sean-k-mooney | yep testing | |