| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-28 | |||
| 09:34:14 | sean-k-mooney | schduler hint ya but that also versioned | |
| 09:34:26 | sean-k-mooney | bauzas: oh for the pack vs spread policy | |
| 09:34:33 | sean-k-mooney | for weighers | |
| 09:34:34 | bauzas | yup | |
| 09:35:06 | sean-k-mooney | ya so i was thining the weigher woudl jsut look at the metadta in the server-group policy | |
| 09:35:18 | sean-k-mooney | and ignore the multiplpere sign | |
| 09:35:31 | sean-k-mooney | or we woudl just set min =0 for the config option | |
| 09:35:47 | bauzas | I'm not in a favor of adding more in the existing server group API but... | |
| 09:35:51 | sean-k-mooney | so the multiplper for the weigher woudl afffect only magnitude | |
| 09:36:07 | bauzas | I guess we need to agree on the use case | |
| 09:36:08 | sean-k-mooney | well server group api is preferable to schduler hit | |
| 09:36:36 | bauzas | sean-k-mooney: from an UX, I think so | |
| 09:36:51 | bauzas | but, you know how much I like our server group implementations | |
| 09:37:54 | bauzas | like --min 2 --hint spread_my_stuff | |
| 09:45:31 | gibi | bauzas: you can look at the soft-anti-affinity weigher for reference how to do it | |
| 09:46:00 | gibi | ie how to do weigher for soft things | |
| 09:47:01 | gibi | jkulik: adding existing instance to server group needs a decision from a) allow adding an instance only if the policy is valid b)allow adding instance even if the policy is not valid yet but will be at the next move | |
| 09:47:28 | gibi | jkulik: I think in the past we was not able to agree on which way we should go | |
| 09:47:32 | gibi | but you can try again | |
| 09:53:40 | jkulik | we've opted for a) ... | |
| 09:53:51 | jkulik | where would be the right point to discuss this? | |
| 09:54:17 | sean-k-mooney | https://github.com/openstack/nova-specs/tree/master/specs | |
| 09:54:31 | sean-k-mooney | sorry my wifi dropped so missed the last few minutes | |
| 09:54:49 | sean-k-mooney | so not sure waht a) is | |
| 09:55:07 | sean-k-mooney | but if you want to creat a spec to discuss the desgin upstream copy https://github.com/openstack/nova-specs/blob/master/specs/zed-template.rst | |
| 09:55:24 | sean-k-mooney | and fill it in and we can discuss on the spec and on irc | |
| 09:56:47 | sean-k-mooney[m] | ah so looking at my matix client i see i did not miss much | |
| 09:56:58 | sean-k-mooney[m] | i dont like multi create | |
| 09:57:04 | sean-k-mooney[m] | so i dont think we should add this tere | |
| 09:58:24 | sean-k-mooney | gibi: correct we did not come to an agreement on how to handel the case where the instance did not comply with the policy | |
| 09:58:44 | sean-k-mooney | i did not like the idea of the add operation implictly live migratinging the instnace | |
| 10:00:29 | sean-k-mooney | we have basicaly 3 options, 1 reject the request if the policy would be violated, 2 accpet but automticaly trigger a move operation to reconsile the state, 3 allow it to be invalide and retturn some kind or warnign and leave it to the end user to reconsile the state with a move operation | |
| 10:00:35 | sean-k-mooney | at a later date | |
| 10:01:11 | jkulik | is moving an instance allowed for normal users? | |
| 10:01:28 | sean-k-mooney | technially yes but only via a resize or shelve | |
| 10:01:42 | sean-k-mooney | live migrate and cold migrate are admin only | |
| 10:11:56 | gibi | sean-k-mooney: yeah, from those options I would go with either reject the add if violates the policy, or accept it but only warn (or extend the server group api to show if the policy is invalid), but I definitely don't want to trigger a move automatically | |
| 10:26:32 | sean-k-mooney | jkulik: gibi this was the previous spec on the topic https://review.opendev.org/c/openstack/nova-specs/+/782353 | |
| 10:28:39 | jkulik | sean-k-mooney: thanks. that will be helpful | |
| 10:29:43 | jkulik | oh, that's proposed as a server-action | |
| 10:33:17 | jkulik | fyi, this is how we built it downstream https://github.com/sapcc/nova/commit/7220be3968ee1dd257c9add88228cc5bb9857795 (+ some commits afterwards to fix certain stuff) | |
| 10:41:19 | sean-k-mooney | i see that still has the same problem | |
| 10:41:42 | frickler | does that ring a bell for someone? "nova.exception.InternalError: Unexpected vif_type=unbound" unstable failure in OSC jobs, shows as JSONDecodeError https://zuul.opendev.org/t/openstack/build/181d8177eab5428a82facc4d95ce00e2 | |
| 10:42:13 | sean-k-mooney | vif_type unbound is what the netron itnerface has before you set the host-id | |
| 10:43:05 | sean-k-mooney | frickler: so } openstackclient.tests.functional.compute.v2.test_server.ServerTests.test_server_attach_detach_floating_ip might be racing | |
| 10:43:09 | sean-k-mooney | with the server boot | |
| 10:43:29 | sean-k-mooney | if it has not finished booting when you try to attach the floating ip then you woudl get that issue i guess | |
| 10:43:56 | frickler | oh, so yet another set of tests needing wait-for-ssh things | |
| 10:44:11 | sean-k-mooney | maybe have not looked at the test yet | |
| 10:44:34 | sean-k-mooney | its not nessiarly sshable | |
| 10:44:38 | sean-k-mooney | it woudl need to be active | |
| 10:45:01 | sean-k-mooney | so this is the test https://github.com/openstack/python-openstackclient/blob/master/openstackclient/tests/functional/compute/v2/test_server.py#L339= | |
| 10:45:16 | sean-k-mooney | it shoud be waiting for active | |
| 10:45:39 | sean-k-mooney | it looks like the error is coming form self.server_create | |
| 10:46:12 | sean-k-mooney | ah from here https://github.com/openstack/python-openstackclient/blob/20e7b01af8f0fb4cf0f4af253270ad470926ba4e/openstackclient/tests/functional/compute/v2/common.py#L89= | |
| 10:46:46 | sean-k-mooney | frickler: so its assuming that a value will be populated in the output i guess | |
| 10:47:28 | sean-k-mooney | from the trace it looks like it got an empty responce of something like that | |
| 10:47:54 | sean-k-mooney | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) | |
| 10:48:42 | sean-k-mooney | frickler: without a request id it might not be easy to see what the api returned | |
| 10:49:39 | sean-k-mooney | 136.246758s that seamed to take a very long time | |
| 10:49:49 | sean-k-mooney | is there a timeout for wait? | |
| 11:15:58 | frickler | neutron says "Concurrent port binding operations failed on port 5b1348e9-4048-4b39-a1df-1161a798052e" before nova fails, so more likely an issue on their side | |
| 11:19:08 | sean-k-mooney | concurrent? | |
| 11:19:13 | sean-k-mooney | how | |
| 11:19:35 | sean-k-mooney | that could only happne if we failed reschulded and tried to bind again | |
| 11:19:47 | sean-k-mooney | but they were sitll binding the port to the first host | |
| 11:20:08 | sean-k-mooney | so i guess yes that woudl be a neutorn issue | |
| 11:20:28 | sean-k-mooney | that is not as far as i am aware one of the exctpiotn they are allowed to raise at teh api level | |
| 11:42:33 | frickler | that's just a warning in the q-svc log and iiuc they retry and succeed after that. but it may be that they send a notification to nova about the first attempt anyway | |
| 11:43:55 | sean-k-mooney | maybe i know ralonsoh has a wip patch for somethign else that i dont think will actully help but a race was mentioned in cotext to that | |
| 11:44:58 | sean-k-mooney | frickler: https://review.opendev.org/c/openstack/neutron/+/846422/3 | |
| 11:45:09 | sean-k-mooney | https://bugs.launchpad.net/neutron/+bug/1979072 | |
| 11:45:17 | sean-k-mooney | that shoudl actully be fixed in nova | |
| 11:45:30 | ralonsoh | sean-k-mooney, right, this is just a WIP patch | |
| 11:45:51 | ralonsoh | trying to address an issue that is on the Neutron side | |
| 11:46:03 | sean-k-mooney | right please dont | |
| 11:46:06 | ralonsoh | however that doesn't address the problem of "concurrent port binding" | |
| 11:46:10 | sean-k-mooney | at least not with a periodic | |
| 11:46:11 | ralonsoh | don't what? | |
| 11:46:18 | sean-k-mooney | try an fix it form neutron | |
| 11:46:45 | ralonsoh | ok but the problem is anytime we have this issue, the bug is bug Neutron | |
| 11:46:50 | sean-k-mooney | we need to fix this form the nova side to avoid posible races between nova and neturon | |
| 11:47:20 | sean-k-mooney | right so there are two ways to fix this in nova. 1 make sure we delete the inactive port bidning when we revert | |
| 11:47:31 | sean-k-mooney | i think we try that already today but it can fial | |
| 11:47:41 | sean-k-mooney | second when we live migrate or try to create a portbidning | |
| 11:47:49 | sean-k-mooney | and it already exits delete and recreate | |
| 11:48:03 | sean-k-mooney | that will prevent this form breaking in the future | |
| 11:48:09 | frickler | the failure in osc isn't related to migration. it happens on initial server create. so I don't understand what could cause the duplicate there | |
| 11:48:37 | sean-k-mooney | if the initall port create faild and we resuchlde then we will try binding it to a second host | |
| 11:48:42 | opendevreview | Sergii Golovatiuk proposed openstack/nova master: Replace "db archive" with "db archive_deleted_raws" https://review.opendev.org/c/openstack/nova/+/847963 | |
| 11:48:48 | sean-k-mooney | we wont create a second binding | |
| 11:48:54 | sean-k-mooney | we will just update the host-id | |
| 11:49:20 | sean-k-mooney | but if neutron is still binding it form the first failed attempet we woudl get a concurrent error | |
| 11:49:39 | sean-k-mooney | frickler: did you check the logs to see if the vm was retired on a second host? | |
| 11:51:55 | frickler | sean-k-mooney: it is a single-node job, I would be surprised if that happened | |
| 11:53:38 | sean-k-mooney | frickler: ok well the only other thing i can think of is the client retry | |
| 11:54:03 | sean-k-mooney | ralonsoh: could this happen if nova retryed creating the port binding because thet inital call timed out | |
| 11:54:08 | sean-k-mooney | i think we try 3 times | |
| 11:54:54 | sean-k-mooney | this being "Concurrent port binding operations failed on port ..." | |
| 12:19:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add regression test for bug 1838392 https://review.opendev.org/c/openstack/nova/+/847965 | |