Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-08
15:22:32 lyarwood so it takes a while, but the CSI driver would start polling Cinder instead of Nova and that has caused various issues
15:22:48 lyarwood because they assumed the attachment in Nova was done at that point
15:23:05 dansmith I'm not sure that
15:23:17 dansmith is reasonable behavior to assume from either 200 or 202 though
15:23:47 dansmith you're giving it attachment information, which it is storing, and that triggers some behavior right?
15:24:21 dansmith like, IIRC, 200 is for "I've stored your object, here's the ID you can use to get it again" and 202 is for "I will store that later, but I can't necessarily give you its identifier yet" right?
15:24:27 sean-k-mooney for a 202 they cannot assuem its complete and should expect to poll nova for the compleation
15:25:05 lyarwood dansmith: yeah but my understanding with 200 was that processing was complete with the return
15:25:08 sean-k-mooney for 200 generally you would assume it blocks until its completed but its valid to retrun before that
15:25:17 lyarwood dansmith: and as sean-k-mooney said, 202 requires additional processing
15:25:34 dansmith I think that's true of the REST object and not necessarily the actions implied from the thing you're storing
15:26:18 sean-k-mooney lyarwood: has the k8s csi plugin be fixed to pool nova by the way
15:26:21 dansmith but again, as sean-k-mooney said, this is really a result of them not reading the docs that say it's async and implying which part of the operation is synchronous from the 200
15:26:26 sean-k-mooney regardeless of if we cahnge the api
15:26:36 lyarwood sean-k-mooney: https://github.com/kubernetes/cloud-provider-openstack/issues/1645 not yet
15:27:28 sean-k-mooney ok but they are aware of it and it will get eventually fixed
15:27:33 lyarwood yup sure, just outlining my motivation to fix this
15:27:51 lyarwood ;)
15:27:58 sean-k-mooney :)
15:28:17 sean-k-mooney i assume we have a customer hitting this so that normally seed up fixing things
15:28:21 dansmith to be clear, making it 202 won't fix this, we're just assuming the human that wrote the code would have assumed they should do something different if the code was 202 right?
15:28:38 lyarwood yup that's the assumption, it wouldn't fix anything in CSI
15:29:01 lyarwood I got triggered by the TODO today and wanted to take a swing a killing it
15:29:06 sean-k-mooney if we wnated to fix CSI issue in nova we would have to block
15:29:11 dansmith so, I mean.. we're like changing something that could easily break other clients that did read the docs, based on the assumption that someone who didn't would have done the right thing if it was different
15:29:14 sean-k-mooney wich i dont think we want to do
15:29:18 dansmith pretty meh :)
15:30:26 bauzas ok, sorry guys, I need to bail out from now
15:31:48 lyarwood bauzas: \o if you want to chat on Monday ping me an email, I should have time to chat once I'm on London
15:31:55 lyarwood in*
15:32:15 bauzas lyarwood: don't worry, I'll use the power of community brainstorm to move on with your spec ;)
15:32:32 dansmith I think we've all put on a few pandemic pounds, but being "on london" would require a substantial increase in waist size, I think
15:33:15 lyarwood dansmith: it has been a rough 16 months, it wouldn't take much more
15:33:24 dansmith heh
22:23:34 opendevreview Julia Kreger proposed openstack/nova master: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/813263
#openstack-nova - 2021-10-09
03:35:26 opendevreview norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220
04:04:16 opendevreview norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220
13:08:56 opendevreview Julia Kreger proposed openstack/nova master: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/813263
#openstack-nova - 2021-10-10
14:19:47 opendevreview norman shen proposed openstack/nova master: Add a timeout for waiting volume snapshot ready https://review.opendev.org/c/openstack/nova/+/813297
#openstack-nova - 2021-10-11
07:45:43 bauzas good morning Nova
07:52:06 opendevreview Dmitriy Rabotyagov proposed openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947
08:24:47 plibeau4 hello, I need your help about this proposal change: https://review.opendev.org/c/openstack/nova/+/803713.
08:50:25 gibi good morning
08:51:03 gibi this week feels to be as crazy as the last was for me. you can ping me but I cannot promise things :/
08:55:43 bauzas plibeau4: ack, will add your change for a review
08:56:09 bauzas thanks for reaching us
09:24:41 plibeau4 thx guys take your time it's just to have your feedback It's already in production on myside :)
09:25:17 plibeau4 and if you have time -> https://review.opendev.org/c/openstack/nova/+/803600
09:26:39 sean-k-mooney to me https://review.opendev.org/c/openstack/nova/+/803713 kind of feels like intoducing a race condetion
09:29:12 sean-k-mooney i guess its not a race as it does not quiesce the disk until after the dev rebase is complete deterministically
09:43:45 sean-k-mooney plibeau4: im not conviced moving the quiece is safe to have in production
09:44:09 sean-k-mooney plibeau4: i think it can lead to data currportion if we are not very careful
09:50:05 gibi sean-k-mooney: re: waiting for vif-plugged during hard reboot. What is the piece of information in the port that can be used to decide if the networking backend sends plug event at plug time?
09:53:35 sean-k-mooney gibi: https://specs.openstack.org/openstack/neutron-specs/specs/train/port-binding-extended-information.html
09:53:45 sean-k-mooney binding_details: {
09:53:47 sean-k-mooney ...
09:53:49 sean-k-mooney "connectivity": "l2",
09:53:51 sean-k-mooney "bound_drivers": {"0": "openvswtich"}
09:53:51 gibi thanks
09:53:53 sean-k-mooney }
09:54:00 sean-k-mooney gibi: we could use the bound_drivers to infer this
09:54:13 sean-k-mooney the spec orginaly had an events section that told use when it sent events
09:54:42 sean-k-mooney that was unfrotunetly removed because we could not agree on how to model that
09:55:12 sean-k-mooney so we would have to create a lookup table in nova and map the driver name to it
09:55:36 sean-k-mooney opendaylight was bind time, opendaylight_v2 is plug time
09:56:36 gibi ack. I might need to create a solution on Pike level downstream hence my continued digging in it
09:56:41 sean-k-mooney the conneectivy field is waht we need to use to know if we can safely allow unadress port too
09:56:45 gibi but if the infor is in train then I cannot use that in Pike
09:56:48 sean-k-mooney if its l2 it does not need an ip
09:57:43 sean-k-mooney gibi: i think a workaround config option to force_rebind_on_reboot is the way to go and backport that
09:57:55 gibi yeah I thinking about a config too
09:58:02 sean-k-mooney there are reasons to have that on master by the way
09:58:11 sean-k-mooney combidnind with unpluging on reboot
09:58:52 sean-k-mooney basically if you make reboot: power off-> unplug-> unbind->rebind->plug->spwan
09:59:21 sean-k-mooney then you can do things like change the firewall driver or maybe even ml2/backend
09:59:22 gibi why do you want to add bind to the picture?
09:59:28 gibi ahh I see
09:59:37 sean-k-mooney currently you have to live migrate
09:59:40 gibi to allow change thing that needs a binding
09:59:53 sean-k-mooney but we have some customer saying they want to do it inplace and are ok with stoping the vm
09:59:58 sean-k-mooney yes
10:00:55 sean-k-mooney basicaly some large customes have said to use we cant live migrate every vm we have but we can stop them cause we run everything in duplex mode and can go simplex for a period of time without issue
10:01:45 gibi OK so there is two parallel direction to pursue. 1) add a workaround flag for something like wait_for_plug_event_at_reboot and 2) force_rebind_on_reboot to allow in place update of binding related data
10:02:06 sean-k-mooney so they are fine with stop(tearing down all networking) and start(setting up all networking) to "migrate" form one firewall driver to another (iptables->ovs) or between mech drvers(ml2/ovs->,l2/ovn)
10:02:31 sean-k-mooney gibi: am yes i guess so
10:02:51 gibi the problem (most probably) on my side is that they do VM reboots in big batches due to upgrade and nova sends a lot of unplug - plug event to ODL which fails to finish setting up OVS before the VM starts requesting IP via DHCP
10:04:08 gibi so if they can set a WA flag in nova that nova should wait for the network-vif-plugged before letting the VM run at reboot would help to control the situation
10:04:28 sean-k-mooney yep
10:04:38 gibi and they need it on Pike :/
10:05:51 gibi I will go with a backportable WA flag
10:07:13 sean-k-mooney ya so on master i think we need to look at how and when we use events and for pike workaround flag is definetly the simplest thing as only the custoemr that is affect can opt in by settting it minimising the risk to others that dont need the change
10:09:44 sean-k-mooney gibi: i had planned to bring up ovn live migration at the ptg(which is affected by the vif-plugged-events) but i might just make tha ta general topic since i think the ovn issue will require ovn changes to fully resolve
10:19:47 jpic hi all, i don't understand what is supposed to keep the websocket between the spice html5 client and the spice server? browsers will kill inactive websockets after 60 secs of inactivity, so i suppose the server is supposed to send ping packets regularily
10:40:28 gibi sean-k-mooney: ack
10:47:40 gibi sean-k-mooney: one more thing, I don't see "bound_drivers" ever got implemented in neutron. Only neturon-lib has the field definition https://review.opendev.org/c/openstack/neutron-lib/+/645288/6/neutron_lib/api/definitions/portbindings.py#55 but neutron does not use that
10:47:50 gibi I only see the connectivity field being implemented
10:47:55 gibi in https://review.opendev.org/c/openstack/neutron/+/645645/
11:05:52 sean-k-mooney i see... am i think it was ment to be implemented at the ml2 plugin level above the drivers but you might be right
11:06:36 sean-k-mooney https://review.opendev.org/c/openstack/neutron-lib/+/645288/6/neutron_lib/api/definitions/portbindings.py added bound_drivers to the extension
11:06:42 sean-k-mooney but im not sure if they have used it

Earlier   Later