| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-08 | |||
| 15:18:13 | sean-k-mooney | lyarwood: when we have done this clean up in the past we have paired it with other changes with the logic being a client tha want to use that new feature via a microversion need to be updated anyway so they can adapt to both change at the same time. | |
| 15:18:21 | dansmith | let me ask this way: what client behavior is going to be wrong if we return 200 that will be right if we return 202? | |
| 15:18:31 | dansmith | if there's something there, then it's worth changing | |
| 15:18:39 | dansmith | sean-k-mooney: right | |
| 15:19:37 | lyarwood | tbh with 200 to 202 it's not our clients that I'm worried about | |
| 15:19:56 | lyarwood | it's other callers outside of them and their understanding of the API | |
| 15:20:21 | dansmith | lyarwood: right, but ... what's the behavior or understanding that will be wrong? | |
| 15:20:22 | lyarwood | the CSI plugin in k8s for OpenStack for example has been misusing it for a while | |
| 15:20:29 | sean-k-mooney | your assumeing they are reading the reponce code but not the docs that say this is an async api | |
| 15:20:43 | lyarwood | dansmith: assuming it's sync and not polling | |
| 15:20:46 | dansmith | meaning, generally 202 means "if you query this thing right after the PUT/POST then you will not see it updated so don't expect it to be" .. is it that? or is it "this operation takes a while?" | |
| 15:21:17 | dansmith | lyarwood: but is the actual API operations sync and just not the attachment? | |
| 15:22:04 | lyarwood | we return after we cast to the compute to do the actual attachment | |
| 15:22:09 | sean-k-mooney | dansmith: i think k8s has been assuming that the attachment was completed when it got the 200 | |
| 15:22:32 | lyarwood | so it takes a while, but the CSI driver would start polling Cinder instead of Nova and that has caused various issues | |
| 15:22:48 | lyarwood | because they assumed the attachment in Nova was done at that point | |
| 15:23:05 | dansmith | I'm not sure that | |
| 15:23:17 | dansmith | is reasonable behavior to assume from either 200 or 202 though | |
| 15:23:47 | dansmith | you're giving it attachment information, which it is storing, and that triggers some behavior right? | |
| 15:24:21 | dansmith | like, IIRC, 200 is for "I've stored your object, here's the ID you can use to get it again" and 202 is for "I will store that later, but I can't necessarily give you its identifier yet" right? | |
| 15:24:27 | sean-k-mooney | for a 202 they cannot assuem its complete and should expect to poll nova for the compleation | |
| 15:25:05 | lyarwood | dansmith: yeah but my understanding with 200 was that processing was complete with the return | |
| 15:25:08 | sean-k-mooney | for 200 generally you would assume it blocks until its completed but its valid to retrun before that | |
| 15:25:17 | lyarwood | dansmith: and as sean-k-mooney said, 202 requires additional processing | |
| 15:25:34 | dansmith | I think that's true of the REST object and not necessarily the actions implied from the thing you're storing | |
| 15:26:18 | sean-k-mooney | lyarwood: has the k8s csi plugin be fixed to pool nova by the way | |
| 15:26:21 | dansmith | but again, as sean-k-mooney said, this is really a result of them not reading the docs that say it's async and implying which part of the operation is synchronous from the 200 | |
| 15:26:26 | sean-k-mooney | regardeless of if we cahnge the api | |
| 15:26:36 | lyarwood | sean-k-mooney: https://github.com/kubernetes/cloud-provider-openstack/issues/1645 not yet | |
| 15:27:28 | sean-k-mooney | ok but they are aware of it and it will get eventually fixed | |
| 15:27:33 | lyarwood | yup sure, just outlining my motivation to fix this | |
| 15:27:51 | lyarwood | ;) | |
| 15:27:58 | sean-k-mooney | :) | |
| 15:28:17 | sean-k-mooney | i assume we have a customer hitting this so that normally seed up fixing things | |
| 15:28:21 | dansmith | to be clear, making it 202 won't fix this, we're just assuming the human that wrote the code would have assumed they should do something different if the code was 202 right? | |
| 15:28:38 | lyarwood | yup that's the assumption, it wouldn't fix anything in CSI | |
| 15:29:01 | lyarwood | I got triggered by the TODO today and wanted to take a swing a killing it | |
| 15:29:06 | sean-k-mooney | if we wnated to fix CSI issue in nova we would have to block | |
| 15:29:11 | dansmith | so, I mean.. we're like changing something that could easily break other clients that did read the docs, based on the assumption that someone who didn't would have done the right thing if it was different | |
| 15:29:14 | sean-k-mooney | wich i dont think we want to do | |
| 15:29:18 | dansmith | pretty meh :) | |
| 15:30:26 | bauzas | ok, sorry guys, I need to bail out from now | |
| 15:31:48 | lyarwood | bauzas: \o if you want to chat on Monday ping me an email, I should have time to chat once I'm on London | |
| 15:31:55 | lyarwood | in* | |
| 15:32:15 | bauzas | lyarwood: don't worry, I'll use the power of community brainstorm to move on with your spec ;) | |
| 15:32:32 | dansmith | I think we've all put on a few pandemic pounds, but being "on london" would require a substantial increase in waist size, I think | |
| 15:33:15 | lyarwood | dansmith: it has been a rough 16 months, it wouldn't take much more | |
| 15:33:24 | dansmith | heh | |
| 22:23:34 | opendevreview | Julia Kreger proposed openstack/nova master: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/813263 | |
| #openstack-nova - 2021-10-09 | |||
| 03:35:26 | opendevreview | norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220 | |
| 04:04:16 | opendevreview | norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220 | |
| 13:08:56 | opendevreview | Julia Kreger proposed openstack/nova master: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/813263 | |
| #openstack-nova - 2021-10-10 | |||
| 14:19:47 | opendevreview | norman shen proposed openstack/nova master: Add a timeout for waiting volume snapshot ready https://review.opendev.org/c/openstack/nova/+/813297 | |
| #openstack-nova - 2021-10-11 | |||
| 07:45:43 | bauzas | good morning Nova | |
| 07:52:06 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947 | |
| 08:24:47 | plibeau4 | hello, I need your help about this proposal change: https://review.opendev.org/c/openstack/nova/+/803713. | |
| 08:50:25 | gibi | good morning | |
| 08:51:03 | gibi | this week feels to be as crazy as the last was for me. you can ping me but I cannot promise things :/ | |
| 08:55:43 | bauzas | plibeau4: ack, will add your change for a review | |
| 08:56:09 | bauzas | thanks for reaching us | |
| 09:24:41 | plibeau4 | thx guys take your time it's just to have your feedback It's already in production on myside :) | |
| 09:25:17 | plibeau4 | and if you have time -> https://review.opendev.org/c/openstack/nova/+/803600 | |
| 09:26:39 | sean-k-mooney | to me https://review.opendev.org/c/openstack/nova/+/803713 kind of feels like intoducing a race condetion | |
| 09:29:12 | sean-k-mooney | i guess its not a race as it does not quiesce the disk until after the dev rebase is complete deterministically | |
| 09:43:45 | sean-k-mooney | plibeau4: im not conviced moving the quiece is safe to have in production | |
| 09:44:09 | sean-k-mooney | plibeau4: i think it can lead to data currportion if we are not very careful | |
| 09:50:05 | gibi | sean-k-mooney: re: waiting for vif-plugged during hard reboot. What is the piece of information in the port that can be used to decide if the networking backend sends plug event at plug time? | |
| 09:53:35 | sean-k-mooney | gibi: https://specs.openstack.org/openstack/neutron-specs/specs/train/port-binding-extended-information.html | |
| 09:53:45 | sean-k-mooney | binding_details: { | |
| 09:53:47 | sean-k-mooney | ... | |
| 09:53:49 | sean-k-mooney | "connectivity": "l2", | |
| 09:53:51 | sean-k-mooney | "bound_drivers": {"0": "openvswtich"} | |
| 09:53:51 | gibi | thanks | |
| 09:53:53 | sean-k-mooney | } | |
| 09:54:00 | sean-k-mooney | gibi: we could use the bound_drivers to infer this | |
| 09:54:13 | sean-k-mooney | the spec orginaly had an events section that told use when it sent events | |
| 09:54:42 | sean-k-mooney | that was unfrotunetly removed because we could not agree on how to model that | |
| 09:55:12 | sean-k-mooney | so we would have to create a lookup table in nova and map the driver name to it | |
| 09:55:36 | sean-k-mooney | opendaylight was bind time, opendaylight_v2 is plug time | |
| 09:56:36 | gibi | ack. I might need to create a solution on Pike level downstream hence my continued digging in it | |
| 09:56:41 | sean-k-mooney | the conneectivy field is waht we need to use to know if we can safely allow unadress port too | |
| 09:56:45 | gibi | but if the infor is in train then I cannot use that in Pike | |
| 09:56:48 | sean-k-mooney | if its l2 it does not need an ip | |
| 09:57:43 | sean-k-mooney | gibi: i think a workaround config option to force_rebind_on_reboot is the way to go and backport that | |
| 09:57:55 | gibi | yeah I thinking about a config too | |
| 09:58:02 | sean-k-mooney | there are reasons to have that on master by the way | |
| 09:58:11 | sean-k-mooney | combidnind with unpluging on reboot | |
| 09:58:52 | sean-k-mooney | basically if you make reboot: power off-> unplug-> unbind->rebind->plug->spwan | |
| 09:59:21 | sean-k-mooney | then you can do things like change the firewall driver or maybe even ml2/backend | |
| 09:59:22 | gibi | why do you want to add bind to the picture? | |
| 09:59:28 | gibi | ahh I see | |
| 09:59:37 | sean-k-mooney | currently you have to live migrate | |
| 09:59:40 | gibi | to allow change thing that needs a binding | |
| 09:59:53 | sean-k-mooney | but we have some customer saying they want to do it inplace and are ok with stoping the vm | |
| 09:59:58 | sean-k-mooney | yes | |
| 10:00:55 | sean-k-mooney | basicaly some large customes have said to use we cant live migrate every vm we have but we can stop them cause we run everything in duplex mode and can go simplex for a period of time without issue | |
| 10:01:45 | gibi | OK so there is two parallel direction to pursue. 1) add a workaround flag for something like wait_for_plug_event_at_reboot and 2) force_rebind_on_reboot to allow in place update of binding related data | |
| 10:02:06 | sean-k-mooney | so they are fine with stop(tearing down all networking) and start(setting up all networking) to "migrate" form one firewall driver to another (iptables->ovs) or between mech drvers(ml2/ovs->,l2/ovn) | |
| 10:02:31 | sean-k-mooney | gibi: am yes i guess so | |
| 10:02:51 | gibi | the problem (most probably) on my side is that they do VM reboots in big batches due to upgrade and nova sends a lot of unplug - plug event to ODL which fails to finish setting up OVS before the VM starts requesting IP via DHCP | |