| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-24 | |||
| 14:42:22 | sean-k-mooney | so if you set that and the vswich has the flow installed the arp will never get to the tor | |
| 14:42:22 | amorin | I am currently facing a race condition in my OpenStack deployment. If I delete a port while the instance was booting (before nova plug the interface), then nova plug the interface and left the interface like this, after a while I have a lot of interface staying on the host | |
| 14:42:37 | zigo | Right. | |
| 14:43:09 | sean-k-mooney | so it will have the opisite effect in that learnign will only happen when you get an non arp packet | |
| 14:43:28 | rouk | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1834048 let me know if more is needed. | |
| 14:43:29 | openstack | Launchpad bug 1834048 in OpenStack Compute (nova) "Nova waits indefinitely on ceph client hangs due to network problems" [Undecided,New] | |
| 14:43:52 | zigo | sean-k-mooney: Ok, thanks. | |
| 14:44:10 | mriedem | amorin: like this? https://review.opendev.org/#/c/663382/ | |
| 14:44:27 | sean-k-mooney | rouk: cool thanks for filing ^ lyarwood you might be able to determing if there is soemthing we can do to fix the ceph issue | |
| 14:44:58 | sean-k-mooney | mriedem: hehe i was looking for that but it might help | |
| 14:45:23 | sean-k-mooney | mriedem: that is specifcaly for the case wehre you delete the instance after teh vm has started running but before we have updated the db | |
| 14:45:33 | sean-k-mooney | i belive there are other case were this can happen too | |
| 14:45:34 | mriedem | dansmith: regarding promoting that revert, slaweq could request it since i wouldn't be surprised if it takes all day to run the nova change through and even then will likely fail on some gate bug | |
| 14:45:44 | sean-k-mooney | that was the one i know was definetly broken | |
| 14:45:45 | amorin | mriedem, sean-k-mooney almost like this | |
| 14:46:09 | dansmith | mriedem: yeah that's what I was thinking.. looking at how many other things have -1s on them, this could take a while | |
| 14:46:17 | dansmith | slaweq: mriedem: is that neutron job in neutron's gate? | |
| 14:46:33 | mriedem | it's voting for neutron so i expect it's in neutron's gate queue yeah | |
| 14:46:34 | sean-k-mooney | dansmith: yes it is | |
| 14:47:02 | dansmith | so anything that has a recent check on it will head to the gate but fail every time, making the gate queue super slow too yeah? | |
| 14:47:05 | amorin | mriedem: something like this: | |
| 14:47:07 | amorin | http://paste.openstack.org/show/753313/ | |
| 14:47:11 | sean-k-mooney | amorin: right in your case you are deleteing the neutorn port | |
| 14:47:19 | amorin | yes, while booting the instance | |
| 14:48:00 | sean-k-mooney | well you should not actully get stale port in that case in ovs | |
| 14:48:11 | mriedem | amorin: so nova plugs the vif, starts waiting for the network-vif-plugged event, the port is then deleted and we don't get the event so we timeout and fail the server build | |
| 14:48:17 | sean-k-mooney | in you logg you have the vif unplug which should clean it up | |
| 14:48:38 | mriedem | sean-k-mooney: pretty sure we don't handle vif unplug and actually unplug | |
| 14:48:44 | mriedem | from out of band network-vif-deleted events | |
| 14:48:57 | sean-k-mooney | mriedem: we dont but we do on instance delete | |
| 14:49:17 | sean-k-mooney | so when we role back the spawn we should call unplug and clean up | |
| 14:49:29 | amorin | I dont have unplug call | |
| 14:49:35 | mriedem | isn't that stephen's patch? | |
| 14:49:53 | sean-k-mooney | no his is for a specifc case were we get an exception form the db | |
| 14:50:02 | sean-k-mooney | in that case we dont call cleanup | |
| 14:50:10 | sean-k-mooney | for the timeout case we do | |
| 14:50:21 | mriedem | https://review.opendev.org/#/c/663382/3/nova/compute/manager.py@2023 | |
| 14:50:52 | mriedem | stephen's patch doesn't have any specific cases in it from what i can tell | |
| 14:51:07 | sean-k-mooney | the downstream bug did | |
| 14:51:14 | mriedem | "If an instance fails to build, which is possible for a variety of reasons, we may end up in a situation where we have remnants of a plugged VIF (typically files) left of the host. This is because we cleanup from the neutron perspective but don't attempt to unplug the VIF, a call which may have many side-effects depending on the VIF driver." | |
| 14:51:17 | amorin | my bad, I have unplug in my logs | |
| 14:51:18 | mriedem | oh well the downstream bug... | |
| 14:52:18 | sean-k-mooney | it should be linkend in the upstream one but its addressing the case of when UnexpectedDeletingTaskStateError is raised | |
| 14:52:52 | mriedem | the rh bz is linked into the lp bug, | |
| 14:53:09 | mriedem | i'm just annoyed that i need to follow the breadcrumb trail of bugs to know wtf is going on in a patch | |
| 14:53:34 | sean-k-mooney | ya it should have been copied into the upstream bug | |
| 14:53:57 | mriedem | amorin: if you have a reproducer of *your* scenario can you apply https://review.opendev.org/#/c/663382/ and see if that resolves it? | |
| 14:54:13 | sean-k-mooney | sorry about that. stephen pick that bug up from me because i didnt have time to start on it two weeks ago | |
| 14:54:20 | amorin | mriedem: yes | |
| 14:54:22 | amorin | will do | |
| 14:55:02 | mriedem | sean-k-mooney: no i'm wrong, the details are in the lp bug, it's fine | |
| 14:56:07 | sean-k-mooney | amorin: by the way stephenfin patch will just prevent new interface form leaking it will not clean up existing stale ports | |
| 14:56:55 | amorin | sean-k-mooney: yes of course :p | |
| 14:57:09 | sean-k-mooney | mriedem: as far as i can tell we could leak ports this way for years so its supper latent | |
| 15:22:52 | Nick_A | How do you set quota defaults for new projects? I did openstack quota set --instances 100 default, but new projects still show the instance limit at 10 | |
| 15:25:00 | mriedem | Nick_A: i suggest you read through https://docs.openstack.org/nova/latest/user/quotas.html and/or https://docs.openstack.org/nova/latest/admin/quotas.html | |
| 15:25:08 | mriedem | and if there is a problem in those docs, then bring it here | |
| 15:26:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: docs: Rewrite host aggregate, availability zone docs https://review.opendev.org/667133 | |
| 15:28:04 | Nick_A | I read through both before asking. Double checked my quota set command matches the docs. | |
| 15:28:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove orphaned comment from _get_group_details https://review.opendev.org/667135 | |
| 15:28:44 | mriedem | Nick_A: check if there are per-project quotas set on the project which would override the defaults | |
| 15:30:03 | openstackgerrit | Merged openstack/python-novaclient master: Add irrelevant files in dsvm job https://review.opendev.org/666217 | |
| 15:30:08 | mriedem | https://docs.openstack.org/nova/latest/user/quotas.html#checking-quota is the thing you want to check | |
| 15:31:58 | mriedem | stephenfin: pretty sure https://review.opendev.org/666217 is broken | |
| 15:33:01 | stephenfin | mriedem: Yup :( I'll submit a fixup | |
| 15:34:21 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Only ignore unit test changes for functional jobs https://review.opendev.org/667147 | |
| 15:35:09 | Nick_A | http://paste.openstack.org/show/OOfHU4BMoigiHYmODPMm/ also set instance quota to 100 for "admin" before creating the new project | |
| 15:35:31 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Only ignore unit test changes for functional jobs https://review.opendev.org/667147 | |
| 15:37:48 | mriedem | Nick_A: I think your usage is wrong here: "openstack quota set --instances 100 default" | |
| 15:37:59 | mriedem | that's setting instances quota to 100 for the "default" project, which doesn't exist | |
| 15:38:27 | mriedem | you should be using "--class default" https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/quota.html#quota-set | |
| 15:38:36 | Nick_A | That's what I thought but that seems to be what the docs indicate | |
| 15:38:58 | mriedem | the osc cli docs aren't clear about what a class is or that "default" is the only class supported by nova... | |
| 15:39:00 | Nick_A | https://docs.openstack.org/nova/latest/admin/quotas.html these specifically | |
| 15:39:22 | mriedem | you mean "2. Update a default value for a new project, for example:" | |
| 15:39:24 | mriedem | yeah that looks busted | |
| 15:39:31 | Nick_A | heh | |
| 15:39:57 | mriedem | "$ openstack quota set --QUOTA_NAME QUOTA_VALUE PROJECT_OR_CLASS" is also misleading | |
| 15:40:03 | mriedem | since if you use a class you need to specify --class | |
| 15:40:15 | mriedem | Nick_A: please hit the bug link at the top of the page and report a nova docs bug | |
| 15:42:03 | Nick_A | # openstack quota set --class --instances 100 default <-- worked thank you | |
| 15:44:47 | mriedem | looks like that's been broken since pike https://review.opendev.org/#/c/490952/4 | |
| 15:45:07 | mriedem | when the admin guide was using the nova cli correctly before the import of the docs and then it was changed to use the osc cli incorrectly | |
| 15:45:16 | mriedem | i didn't notice that change since it wasn't mentioned in the commit message.... | |
| 15:45:33 | openstackgerrit | Merged openstack/nova master: Ignore hw_vif_type for direct, direct-physical vNIC types https://review.opendev.org/609460 | |
| 15:45:59 | Nick_A | bug submitted - thank you | |
| 15:47:30 | mriedem | gdi that whole page probably has to be validated now | |
| 15:47:48 | mriedem | Nick_A: thanks | |
| 15:48:15 | mriedem | stephenfin: maybe you want to fix this https://bugs.launchpad.net/nova/+bug/1834057 | |
| 15:48:17 | openstack | Launchpad bug 1834057 in OpenStack Compute (nova) "change default quota command example is incorrect" [Undecided,New] | |
| 15:48:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Unplug VIFs as part of cleanup of networks https://review.opendev.org/663382 | |
| 15:48:30 | mriedem | since you imported those docs and changed the cli used to be wrong w/o mentioning it in the commit | |
| 15:51:28 | stephenfin | mriedem: https://github.com/openstack/openstack-manuals/commit/655a9ce2c6372b94d2023242292aaae3a511e631 | |
| 15:53:16 | mriedem | that's not what's in https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/admin-guide/source/cli-set-compute-quotas.rst | |
| 15:58:12 | stephenfin | You're talking about the footer, I assume. I'm guessing that's because I wanted some kind of semi-permanent record though not the exact one | |
| 15:58:19 | stephenfin | But idk what I was thinking nearly two years ago | |
| 15:59:21 | dansmith | that neutron event patch is making it through check pretty well, compared to how it's been the last few weeks, so maybe not quite as long to gate as we thought | |
| 15:59:30 | stephenfin | Point is, I didn't change anything to be wrong. You could have just asked me to investigate and fix without assigning blame | |
| 16:05:47 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Revert "Add irrelevant files in dsvm job" https://review.opendev.org/667151 | |
| 16:06:55 | mriedem | stephenfin: yes the footer in the commit message points at the docs from stable/ocata so that's what i assumed you imported from, and those ocata docs were using nova CLIs but your change was not, so i assumed you changed things while importing | |