| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-24 | |||
| 14:21:44 | dansmith | mriedem: okay I saw you said you were "OK" with it, but didn't see that was the plan | |
| 14:21:51 | dansmith | so let's update it for him and approve? | |
| 14:21:57 | mriedem | yeah i can do that | |
| 14:22:15 | sean-k-mooney | dansmith: ya we are going to revert | |
| 14:23:11 | openstackgerrit | Slawek Kaplonski proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:23:16 | sean-k-mooney | to be honest this give me more time to test this for the same host resize case too and the backport will be simpler since it will be 1 patch so it better in the log run | |
| 14:23:44 | dansmith | mriedem: ^ good? | |
| 14:24:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 14:24:23 | mriedem | d'oh | |
| 14:25:21 | mriedem | dansmith: +2 | |
| 14:26:00 | dansmith | okay cool | |
| 14:29:09 | slaweq | mriedem: hi, I just updated this commit message, sorry for the delay, I was on meeting | |
| 14:29:10 | mriedem | sean-k-mooney: i've marked the original bug as "triaged" rather than "fix released" so your patch can still close the original bug https://bugs.launchpad.net/nova/+bug/1832028 | |
| 14:29:11 | openstack | Launchpad bug 1832028 in OpenStack Compute (nova) stein "revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 14:29:29 | mriedem | slaweq: yup me too :) the revert is approved | |
| 14:29:36 | slaweq | mriedem: thx a lot | |
| 14:29:49 | mriedem | slaweq: i'll just say again that the inconsistency of when neutron sends these events per vif type sucks | |
| 14:29:52 | mriedem | from a consumer pov | |
| 14:30:23 | slaweq | mriedem: yes, I know | |
| 14:30:49 | slaweq | we also found other issue with live-migration and e.g. networking-ovn recently but that will be reported separately later | |
| 14:31:08 | mriedem | i've heard (from the rh nova team) | |
| 14:31:17 | mriedem | which is why you changed to ovs hybrid plug and then found this issue | |
| 14:31:18 | sean-k-mooney | mriedem: ack, do i need to respond on the ml | |
| 14:32:01 | mriedem | sean-k-mooney: saying we approved the revert on the ML is probably good for people to hold their rechecks | |
| 14:33:06 | dansmith | promotion time? | |
| 14:33:30 | mriedem | i'm not the person to ask probably | |
| 14:33:44 | dansmith | sean-k-mooney: lol | |
| 14:33:48 | mriedem | it won't block nova of course | |
| 14:33:57 | mriedem | we're just blocked by the rate limiting | |
| 14:34:23 | sean-k-mooney | mriedem: on the upside since neutron are not using there gate currently nova should speed up :P | |
| 14:36:55 | zigo | Is there an openstackclient equivalent to nova server-migration-show? | |
| 14:37:08 | amorin | hey all | |
| 14:38:21 | sean-k-mooney | zigo: not that i see at a glance | |
| 14:38:41 | zigo | sean-k-mooney: Ok, so I'm not completely bind then! :) | |
| 14:39:06 | stephenfin | zigo: I don't think so. That was one of the gaps we identified in Denver (summit) | |
| 14:39:09 | sean-k-mooney | you are not :) we dont have full parity | |
| 14:39:36 | sean-k-mooney | zigo: initally osc did not have admin commnds so that is the area we have the most gaps today | |
| 14:39:46 | zigo | Right. | |
| 14:40:46 | zigo | If I put arp_responder = True, will this speed-up MAC learning for the network equipment connected to my compute? | |
| 14:41:07 | zigo | It usually takes dozens of seconds for it to learn what the new floating IP location is... | |
| 14:41:09 | sean-k-mooney | no | |
| 14:41:21 | zigo | sean-k-mooney: Is there some tricks? :) | |
| 14:42:04 | sean-k-mooney | arp_resopnder = true need the l2_population ml2 driver and does arp suppression so that openflow rules rewrite the arp request to an arp responce | |
| 14:42:22 | amorin | I am currently facing a race condition in my OpenStack deployment. If I delete a port while the instance was booting (before nova plug the interface), then nova plug the interface and left the interface like this, after a while I have a lot of interface staying on the host | |
| 14:42:22 | sean-k-mooney | so if you set that and the vswich has the flow installed the arp will never get to the tor | |
| 14:42:37 | zigo | Right. | |
| 14:43:09 | sean-k-mooney | so it will have the opisite effect in that learnign will only happen when you get an non arp packet | |
| 14:43:28 | rouk | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1834048 let me know if more is needed. | |
| 14:43:29 | openstack | Launchpad bug 1834048 in OpenStack Compute (nova) "Nova waits indefinitely on ceph client hangs due to network problems" [Undecided,New] | |
| 14:43:52 | zigo | sean-k-mooney: Ok, thanks. | |
| 14:44:10 | mriedem | amorin: like this? https://review.opendev.org/#/c/663382/ | |
| 14:44:27 | sean-k-mooney | rouk: cool thanks for filing ^ lyarwood you might be able to determing if there is soemthing we can do to fix the ceph issue | |
| 14:44:58 | sean-k-mooney | mriedem: hehe i was looking for that but it might help | |
| 14:45:23 | sean-k-mooney | mriedem: that is specifcaly for the case wehre you delete the instance after teh vm has started running but before we have updated the db | |
| 14:45:33 | sean-k-mooney | i belive there are other case were this can happen too | |
| 14:45:34 | mriedem | dansmith: regarding promoting that revert, slaweq could request it since i wouldn't be surprised if it takes all day to run the nova change through and even then will likely fail on some gate bug | |
| 14:45:44 | sean-k-mooney | that was the one i know was definetly broken | |
| 14:45:45 | amorin | mriedem, sean-k-mooney almost like this | |
| 14:46:09 | dansmith | mriedem: yeah that's what I was thinking.. looking at how many other things have -1s on them, this could take a while | |
| 14:46:17 | dansmith | slaweq: mriedem: is that neutron job in neutron's gate? | |
| 14:46:33 | mriedem | it's voting for neutron so i expect it's in neutron's gate queue yeah | |
| 14:46:34 | sean-k-mooney | dansmith: yes it is | |
| 14:47:02 | dansmith | so anything that has a recent check on it will head to the gate but fail every time, making the gate queue super slow too yeah? | |
| 14:47:05 | amorin | mriedem: something like this: | |
| 14:47:07 | amorin | http://paste.openstack.org/show/753313/ | |
| 14:47:11 | sean-k-mooney | amorin: right in your case you are deleteing the neutorn port | |
| 14:47:19 | amorin | yes, while booting the instance | |
| 14:48:00 | sean-k-mooney | well you should not actully get stale port in that case in ovs | |
| 14:48:11 | mriedem | amorin: so nova plugs the vif, starts waiting for the network-vif-plugged event, the port is then deleted and we don't get the event so we timeout and fail the server build | |
| 14:48:17 | sean-k-mooney | in you logg you have the vif unplug which should clean it up | |
| 14:48:38 | mriedem | sean-k-mooney: pretty sure we don't handle vif unplug and actually unplug | |
| 14:48:44 | mriedem | from out of band network-vif-deleted events | |
| 14:48:57 | sean-k-mooney | mriedem: we dont but we do on instance delete | |
| 14:49:17 | sean-k-mooney | so when we role back the spawn we should call unplug and clean up | |
| 14:49:29 | amorin | I dont have unplug call | |
| 14:49:35 | mriedem | isn't that stephen's patch? | |
| 14:49:53 | sean-k-mooney | no his is for a specifc case were we get an exception form the db | |
| 14:50:02 | sean-k-mooney | in that case we dont call cleanup | |
| 14:50:10 | sean-k-mooney | for the timeout case we do | |
| 14:50:21 | mriedem | https://review.opendev.org/#/c/663382/3/nova/compute/manager.py@2023 | |
| 14:50:52 | mriedem | stephen's patch doesn't have any specific cases in it from what i can tell | |
| 14:51:07 | sean-k-mooney | the downstream bug did | |
| 14:51:14 | mriedem | "If an instance fails to build, which is possible for a variety of reasons, we may end up in a situation where we have remnants of a plugged VIF (typically files) left of the host. This is because we cleanup from the neutron perspective but don't attempt to unplug the VIF, a call which may have many side-effects depending on the VIF driver." | |
| 14:51:17 | amorin | my bad, I have unplug in my logs | |
| 14:51:18 | mriedem | oh well the downstream bug... | |
| 14:52:18 | sean-k-mooney | it should be linkend in the upstream one but its addressing the case of when UnexpectedDeletingTaskStateError is raised | |
| 14:52:52 | mriedem | the rh bz is linked into the lp bug, | |
| 14:53:09 | mriedem | i'm just annoyed that i need to follow the breadcrumb trail of bugs to know wtf is going on in a patch | |
| 14:53:34 | sean-k-mooney | ya it should have been copied into the upstream bug | |
| 14:53:57 | mriedem | amorin: if you have a reproducer of *your* scenario can you apply https://review.opendev.org/#/c/663382/ and see if that resolves it? | |
| 14:54:13 | sean-k-mooney | sorry about that. stephen pick that bug up from me because i didnt have time to start on it two weeks ago | |
| 14:54:20 | amorin | mriedem: yes | |
| 14:54:22 | amorin | will do | |
| 14:55:02 | mriedem | sean-k-mooney: no i'm wrong, the details are in the lp bug, it's fine | |
| 14:56:07 | sean-k-mooney | amorin: by the way stephenfin patch will just prevent new interface form leaking it will not clean up existing stale ports | |
| 14:56:55 | amorin | sean-k-mooney: yes of course :p | |
| 14:57:09 | sean-k-mooney | mriedem: as far as i can tell we could leak ports this way for years so its supper latent | |
| 15:22:52 | Nick_A | How do you set quota defaults for new projects? I did openstack quota set --instances 100 default, but new projects still show the instance limit at 10 | |
| 15:25:00 | mriedem | Nick_A: i suggest you read through https://docs.openstack.org/nova/latest/user/quotas.html and/or https://docs.openstack.org/nova/latest/admin/quotas.html | |
| 15:25:08 | mriedem | and if there is a problem in those docs, then bring it here | |