| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-28 | |||
| 09:26:04 | sean-k-mooney | so there soft vs hard "requirement" would be filter vs weigher | |
| 09:26:20 | sean-k-mooney | but most of the logic would be the same so i woudl proably just do both | |
| 09:26:23 | jkulik | sean-k-mooney: I don't think I can make it in 2 weeks, but would be able to do it in the longer run | |
| 09:26:26 | sean-k-mooney | and try and shre the code | |
| 09:26:51 | sean-k-mooney | jkulik: ack | |
| 09:28:40 | jkulik | custom scheduler filter would be my idea, too. but we'd need it in the server-group API anyways as having one flavor per rack really doesn't scale | |
| 09:29:29 | sean-k-mooney | not per rack | |
| 09:29:36 | sean-k-mooney | so in the flavor you would have a policy | |
| 09:29:54 | sean-k-mooney | aggreate_anti_affinity:rack | |
| 09:30:12 | sean-k-mooney | aggreate_anti_affinity:room | |
| 09:30:26 | sean-k-mooney | aggreate_anti_affinity:row | |
| 09:30:30 | jkulik | ah, but still. customers are already overwhelmed by the number of flavors as is. I don't think we can add more for this :) | |
| 09:30:57 | sean-k-mooney | ya that is why the server-group api woudl be beeter | |
| 09:31:18 | sean-k-mooney | ther is a hack that you could use if you promise not to mention my name as the source :P | |
| 09:31:26 | jkulik | :D | |
| 09:31:37 | sean-k-mooney | so you could use server tags or instance metadta for now | |
| 09:31:42 | bauzas | I wonder how feasible a weigher could be | |
| 09:32:15 | bauzas | we could weigh per aggregatesd | |
| 09:32:17 | sean-k-mooney | so you could add aggreate_anti_affinity_* set of server tags | |
| 09:32:32 | sean-k-mooney | and then look at those in the filter/weigher | |
| 09:32:34 | bauzas | each host getting the same weigh | |
| 09:32:49 | jkulik | sean-k-mooney: that sounds wrong :D but thanks for the idea | |
| 09:32:51 | sean-k-mooney | bauzas: that is doable yes | |
| 09:33:32 | sean-k-mooney | jkulik: tags are somethimes used for this in other out of tree implemenation like tripleos instance ha feature | |
| 09:33:35 | jkulik | speaking of the server-group API: we've extended it to allow adding servers to server-groups after they're spawned. this can be helpful if you need to spawn a new instance anti-affin to a previously spawned one, where you didn't know that requirement, yet. | |
| 09:33:42 | bauzas | I'm still confused by the weights | |
| 09:33:48 | sean-k-mooney | its not how the api is inteded to be used but still | |
| 09:33:48 | bauzas | but | |
| 09:33:52 | jkulik | is that something that has a chance upstream, if we write a spec for it? | |
| 09:33:56 | bauzas | you could pass a hint | |
| 09:34:09 | bauzas | and then have a weigher looking up at the hint | |
| 09:34:14 | sean-k-mooney | schduler hint ya but that also versioned | |
| 09:34:26 | sean-k-mooney | bauzas: oh for the pack vs spread policy | |
| 09:34:33 | sean-k-mooney | for weighers | |
| 09:34:34 | bauzas | yup | |
| 09:35:06 | sean-k-mooney | ya so i was thining the weigher woudl jsut look at the metadta in the server-group policy | |
| 09:35:18 | sean-k-mooney | and ignore the multiplpere sign | |
| 09:35:31 | sean-k-mooney | or we woudl just set min =0 for the config option | |
| 09:35:47 | bauzas | I'm not in a favor of adding more in the existing server group API but... | |
| 09:35:51 | sean-k-mooney | so the multiplper for the weigher woudl afffect only magnitude | |
| 09:36:07 | bauzas | I guess we need to agree on the use case | |
| 09:36:08 | sean-k-mooney | well server group api is preferable to schduler hit | |
| 09:36:36 | bauzas | sean-k-mooney: from an UX, I think so | |
| 09:36:51 | bauzas | but, you know how much I like our server group implementations | |
| 09:37:54 | bauzas | like --min 2 --hint spread_my_stuff | |
| 09:45:31 | gibi | bauzas: you can look at the soft-anti-affinity weigher for reference how to do it | |
| 09:46:00 | gibi | ie how to do weigher for soft things | |
| 09:47:01 | gibi | jkulik: adding existing instance to server group needs a decision from a) allow adding an instance only if the policy is valid b)allow adding instance even if the policy is not valid yet but will be at the next move | |
| 09:47:28 | gibi | jkulik: I think in the past we was not able to agree on which way we should go | |
| 09:47:32 | gibi | but you can try again | |
| 09:53:40 | jkulik | we've opted for a) ... | |
| 09:53:51 | jkulik | where would be the right point to discuss this? | |
| 09:54:17 | sean-k-mooney | https://github.com/openstack/nova-specs/tree/master/specs | |
| 09:54:31 | sean-k-mooney | sorry my wifi dropped so missed the last few minutes | |
| 09:54:49 | sean-k-mooney | so not sure waht a) is | |
| 09:55:07 | sean-k-mooney | but if you want to creat a spec to discuss the desgin upstream copy https://github.com/openstack/nova-specs/blob/master/specs/zed-template.rst | |
| 09:55:24 | sean-k-mooney | and fill it in and we can discuss on the spec and on irc | |
| 09:56:47 | sean-k-mooney[m] | ah so looking at my matix client i see i did not miss much | |
| 09:56:58 | sean-k-mooney[m] | i dont like multi create | |
| 09:57:04 | sean-k-mooney[m] | so i dont think we should add this tere | |
| 09:58:24 | sean-k-mooney | gibi: correct we did not come to an agreement on how to handel the case where the instance did not comply with the policy | |
| 09:58:44 | sean-k-mooney | i did not like the idea of the add operation implictly live migratinging the instnace | |
| 10:00:29 | sean-k-mooney | we have basicaly 3 options, 1 reject the request if the policy would be violated, 2 accpet but automticaly trigger a move operation to reconsile the state, 3 allow it to be invalide and retturn some kind or warnign and leave it to the end user to reconsile the state with a move operation | |
| 10:00:35 | sean-k-mooney | at a later date | |
| 10:01:11 | jkulik | is moving an instance allowed for normal users? | |
| 10:01:28 | sean-k-mooney | technially yes but only via a resize or shelve | |
| 10:01:42 | sean-k-mooney | live migrate and cold migrate are admin only | |
| 10:11:56 | gibi | sean-k-mooney: yeah, from those options I would go with either reject the add if violates the policy, or accept it but only warn (or extend the server group api to show if the policy is invalid), but I definitely don't want to trigger a move automatically | |
| 10:26:32 | sean-k-mooney | jkulik: gibi this was the previous spec on the topic https://review.opendev.org/c/openstack/nova-specs/+/782353 | |
| 10:28:39 | jkulik | sean-k-mooney: thanks. that will be helpful | |
| 10:29:43 | jkulik | oh, that's proposed as a server-action | |
| 10:33:17 | jkulik | fyi, this is how we built it downstream https://github.com/sapcc/nova/commit/7220be3968ee1dd257c9add88228cc5bb9857795 (+ some commits afterwards to fix certain stuff) | |
| 10:41:19 | sean-k-mooney | i see that still has the same problem | |
| 10:41:42 | frickler | does that ring a bell for someone? "nova.exception.InternalError: Unexpected vif_type=unbound" unstable failure in OSC jobs, shows as JSONDecodeError https://zuul.opendev.org/t/openstack/build/181d8177eab5428a82facc4d95ce00e2 | |
| 10:42:13 | sean-k-mooney | vif_type unbound is what the netron itnerface has before you set the host-id | |
| 10:43:05 | sean-k-mooney | frickler: so } openstackclient.tests.functional.compute.v2.test_server.ServerTests.test_server_attach_detach_floating_ip might be racing | |
| 10:43:09 | sean-k-mooney | with the server boot | |
| 10:43:29 | sean-k-mooney | if it has not finished booting when you try to attach the floating ip then you woudl get that issue i guess | |
| 10:43:56 | frickler | oh, so yet another set of tests needing wait-for-ssh things | |
| 10:44:11 | sean-k-mooney | maybe have not looked at the test yet | |
| 10:44:34 | sean-k-mooney | its not nessiarly sshable | |
| 10:44:38 | sean-k-mooney | it woudl need to be active | |
| 10:45:01 | sean-k-mooney | so this is the test https://github.com/openstack/python-openstackclient/blob/master/openstackclient/tests/functional/compute/v2/test_server.py#L339= | |
| 10:45:16 | sean-k-mooney | it shoud be waiting for active | |
| 10:45:39 | sean-k-mooney | it looks like the error is coming form self.server_create | |
| 10:46:12 | sean-k-mooney | ah from here https://github.com/openstack/python-openstackclient/blob/20e7b01af8f0fb4cf0f4af253270ad470926ba4e/openstackclient/tests/functional/compute/v2/common.py#L89= | |
| 10:46:46 | sean-k-mooney | frickler: so its assuming that a value will be populated in the output i guess | |
| 10:47:28 | sean-k-mooney | from the trace it looks like it got an empty responce of something like that | |
| 10:47:54 | sean-k-mooney | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) | |
| 10:48:42 | sean-k-mooney | frickler: without a request id it might not be easy to see what the api returned | |
| 10:49:39 | sean-k-mooney | 136.246758s that seamed to take a very long time | |
| 10:49:49 | sean-k-mooney | is there a timeout for wait? | |
| 11:15:58 | frickler | neutron says "Concurrent port binding operations failed on port 5b1348e9-4048-4b39-a1df-1161a798052e" before nova fails, so more likely an issue on their side | |
| 11:19:08 | sean-k-mooney | concurrent? | |
| 11:19:13 | sean-k-mooney | how | |
| 11:19:35 | sean-k-mooney | that could only happne if we failed reschulded and tried to bind again | |
| 11:19:47 | sean-k-mooney | but they were sitll binding the port to the first host | |
| 11:20:08 | sean-k-mooney | so i guess yes that woudl be a neutorn issue | |
| 11:20:28 | sean-k-mooney | that is not as far as i am aware one of the exctpiotn they are allowed to raise at teh api level | |
| 11:42:33 | frickler | that's just a warning in the q-svc log and iiuc they retry and succeed after that. but it may be that they send a notification to nova about the first attempt anyway | |