| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-31 | |||
| 21:21:51 | pramodrj07 | Can anyone help me in understanding the differnece between security groups in nova and sg in neutron? | |
| 21:22:51 | mriedem | pramodrj07: security groups in nova are tied to nova-network, which has been deprecated since the newton release | |
| 21:23:17 | pramodrj07 | Oh cool! Thanks for clarification | |
| 21:23:26 | mriedem | the security group resource APIs were deprecated in the 2.36 microversion | |
| 21:24:14 | mriedem | you can provide security groups when creating a server, and nova will associate the security groups with ports that are attached to the server when creating it | |
| 21:24:33 | mriedem | assuming the network that the port is on enables port security | |
| 21:25:04 | pramodrj07 | Oh ok! So the latest cli client will call neutron security group API when we try to create a sg!? | |
| 21:26:27 | mriedem | which cli? nova boot or openstack server create? | |
| 21:26:45 | pramodrj07 | openstack security group create | |
| 21:26:55 | pramodrj07 | yes Openstack one | |
| 21:27:03 | mriedem | that is openstackclient and will use neutron if there is a network service type in the service catalog | |
| 21:28:57 | pramodrj07 | Oh ok! Gotcha! | |
| 21:30:49 | pramodrj07 | i am extending a plugin for rally to create sgs along with sg rules for neutron. I saw a similar scenario developed for nova. So wanted a clarification! | |
| 21:30:58 | pramodrj07 | Thanks a lot mriedem | |
| 21:31:47 | mriedem | yw | |
| 22:40:23 | openstack | Launchpad bug 1678326 in OpenStack Compute (nova) "Quota is decremented during instance delete in cell0 even if the instance destroy fails" [High,Triaged] - Assigned to Matt Riedemann (mriedem) | |
| 22:40:23 | mriedem | melwitt: here is the bug for the quota commit/rollback thing in the api https://bugs.launchpad.net/nova/+bug/1678326 | |
| 22:40:28 | mriedem | long-time coming on writing that | |
| 22:41:31 | mriedem | i don't have a good way of writing a regression test for that one really, unless i stub out instance.destroy() to raise InstanceNotFound to simulate concurrent deletes | |
| 22:41:43 | melwitt | mriedem: thanks for writing that up | |
| 22:42:38 | melwitt | I'm trying to remember how does the instance destroy fail | |
| 22:43:20 | mriedem | it could happen with concurrent delete requests couldn't it? | |
| 22:43:56 | melwitt | I just can't remember what happened to where we figured this out | |
| 22:46:04 | mriedem | idk, it was copied from _delete_while_booting | |
| 22:46:15 | mriedem | which was copied from _delete | |
| 22:46:26 | mriedem | which checks for that here https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1806 | |
| 22:47:56 | mriedem | sorry wrong link there | |
| 22:48:01 | openstack | Launchpad bug 1670627 in OpenStack Compute (nova) ocata "quota is always in-use after delete the ERROR instances " [Critical,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 22:48:01 | melwitt | looks like int32bit described it in a comment on https://bugs.launchpad.net/nova/+bug/1670627 but didn't say how delete failed | |
| 22:48:09 | mriedem | https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L2025 | |
| 22:48:59 | melwitt | sorry, krims0n32 is the lp name | |
| 22:49:08 | mriedem | yeah i was wondering about that | |
| 22:49:34 | mriedem | anyway, the old old code just says # NOTE(comstud): Race condition. Instance already gone. | |
| 22:49:38 | mriedem | again, this is all comstud's fault | |
| 22:49:55 | openstack | Launchpad bug 1670627 in OpenStack Compute (nova) ocata "quota is always in-use after delete the ERROR instances " [Critical,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 22:49:55 | melwitt | https://bugs.launchpad.net/nova/+bug/1670627/comments/27 | |
| 22:50:05 | melwitt | heh | |
| 22:50:34 | mriedem | 2012 so who knows https://github.com/openstack/nova/commit/c729ba8c0aa4d283e84d139bc98e0e89fd933c4a | |
| 22:50:45 | mriedem | seemed like a good idea at the time | |
| 22:51:16 | mriedem | have you ever noticed this? https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L2010 | |
| 22:51:34 | mriedem | i bet ^ is wrong now with the cellsv2 local delete case too | |
| 22:51:39 | melwitt | yeah, I was just curious how it happened, mostly to make sure that's the only problem and there isn't some additional problem | |
| 22:52:35 | melwitt | yeah, I have seen that before but it gets glossed over bc soft-delete | |
| 22:53:14 | melwitt | it gets removed in my series (all reserve/commit/rollback removed) | |
| 22:53:41 | mriedem | i noticed it because of this https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/manager.py#L2334 | |
| 22:53:55 | mriedem | the dumb thing is that we create the reservation in the api then just roll it back right away | |
| 22:54:05 | mriedem | we'd be smarter to just not create the reservations in the first place if it's soft delete | |
| 22:54:20 | mriedem | but i'm sure that was fixed as a bolt-on bug fix to the already 30 bolt-ons to the local delete code | |
| 22:54:28 | melwitt | yeah, for real | |
| 22:56:14 | mriedem | ok so maybe rather than me spending time making quotas.commit/rollback fail instead of noop if there are no reservations, because of counting quotas removing all of that in master, i can just work on the functional regression test recreate and the fix since those have to be backported to ocata | |
| 22:56:55 | openstack | Launchpad bug 1675570 in OpenStack Compute (nova) "Volumes are not detached when deleting shelved offloaded servers" [High,In progress] - Assigned to Zhenyu Zheng (zhengzhenyu) | |
| 22:56:55 | mriedem | i wanted to fix this before https://bugs.launchpad.net/nova/+bug/1675570 | |
| 22:58:13 | melwitt | yeah. I wonder if some of that will get consolidated bc of the need to call _local_delete in some cases | |
| 22:58:23 | mriedem | that's my plan | |
| 22:58:34 | mriedem | rather than duplicate/triplicate everything, just re-use _local_delete | |
| 22:58:37 | mriedem | for the latter bug | |
| 22:59:15 | melwitt | _local_delete is needed if the instance had ever been scheduled. which you can't tell from only instance.host bc of shelved offloaded I guess | |
| 23:00:04 | melwitt | the other case (the 5-time-reverted patch I have up) is an instance in error state. it could have volumes attached even though it has instance.host = None | |
| 23:00:34 | melwitt | because there's code in compute that resets instance.host to None when setting to ERROR state | |
| 23:00:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Make quotas.commit/rollback strict https://review.openstack.org/452346 | |
| 23:25:15 | nrezinorn | How new is the server.create_image snapshot function in nova client/api? I am seeing a problem where randomly the snapshot fails, and the VM stays powered off. | |
| 23:28:11 | mriedem | nrezinorn: probably extremely old | |
| 23:28:14 | mriedem | git blame will tell you | |
| 23:28:30 | mriedem | the client side code shouldn't be causing random issues, you're likely hitting something on the server side | |
| 23:28:47 | nrezinorn | its definately stuff server side, as you can see the errors here | |
| 23:28:55 | nrezinorn | https://gist.github.com/Nrezinorn/decc29d0f142eee34c693e7220090ad9 | |
| 23:29:29 | mriedem | qemu is blowing up | |
| 23:29:30 | nrezinorn | ^in that i took a snapshot fine on that VM ~ 20-30 prior and it worked fine | |
| 23:29:50 | nrezinorn | it doesnt always blow up. | |
| 23:31:01 | nrezinorn | i will look through more RH bugreports for qemu , is there anything i can enable on the hupervisor to get more info than the error message from nova-compute? | |
| 23:31:12 | mriedem | libvirt debug logging | |
| 23:31:20 | mriedem | sec | |
| 23:31:43 | mriedem | this is how we setup libvirt debug logging for our CI runs https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/functions-libvirt#L104 | |
| 23:32:01 | mriedem | what version of qemu/libvirt? | |
| 23:33:14 | nrezinorn | libvirt-2.0.0-10.el7_3.4.x86_64 qemu-kvm-1.5.3-126.el7_3.3.x86_64 | |
| 23:34:24 | mriedem | hmm, idk then, we're using what's in ubuntu 16.04 right now which is libvirt 1.3.1 | |
| 23:34:41 | nrezinorn | sorry this is on centos ;) | |
| 23:34:56 | nrezinorn | and i already looked to see if there were updated packages for both qemu and libvirt | |
| 23:35:09 | mriedem | that qemu looks pretty old actually | |
| 23:35:12 | mriedem | https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix | |
| 23:35:12 | nrezinorn | i will attempt the debug stuff you link on monday to see what is really going on. | |
| 23:35:26 | nrezinorn | pretty old == CentOS7 pretty old ;) | |
| 23:35:32 | mriedem | 7.3 2016-11-03 2.0.0 2.6.0 | |
| 23:35:36 | mriedem | is it centos 7.3? | |
| 23:35:39 | nrezinorn | yes | |
| 23:35:40 | mriedem | oh 7.0 | |
| 23:35:51 | mriedem | if it's 7.3, ^ says you should have qemu 2.6.0 | |
| 23:35:56 | mriedem | not 1.5.3 | |
| 23:36:21 | nrezinorn | hmmm | |
| 23:36:23 | nrezinorn | ok thanks | |
| 23:36:26 | mriedem | np | |
| 23:36:42 | nrezinorn | i will look into why the packages are not lining up :) | |
| 23:38:32 | nrezinorn | hmm, wonder where that 2.6.0 rpm lives...on my machine im on now its 7.3 and i only see 1.5.3 like above | |
| 23:44:07 | mriedem | melwitt: ah writing a regression test for this commit/rollback thing shows me yet another problem :) | |
| 23:44:12 | mriedem | it's a bountiful harvest of bugs | |
| 23:44:21 | melwitt | :( | |
| 23:45:10 | mriedem | nrezinorn: i'm not sure which service channels are used to build that list, or if it's just stock (i assume it's supposed to be stock) | |
| 23:45:34 | mriedem | you could ask sgordon but i think it's too late in the day for his tz | |
| 23:46:41 | mriedem | melwitt: right here https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1891 | |
| 23:46:46 | mriedem | we don't return | |