| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-31 | |||
| 22:49:08 | mriedem | yeah i was wondering about that | |
| 22:49:34 | mriedem | anyway, the old old code just says # NOTE(comstud): Race condition. Instance already gone. | |
| 22:49:38 | mriedem | again, this is all comstud's fault | |
| 22:49:55 | openstack | Launchpad bug 1670627 in OpenStack Compute (nova) ocata "quota is always in-use after delete the ERROR instances " [Critical,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 22:49:55 | melwitt | https://bugs.launchpad.net/nova/+bug/1670627/comments/27 | |
| 22:50:05 | melwitt | heh | |
| 22:50:34 | mriedem | 2012 so who knows https://github.com/openstack/nova/commit/c729ba8c0aa4d283e84d139bc98e0e89fd933c4a | |
| 22:50:45 | mriedem | seemed like a good idea at the time | |
| 22:51:16 | mriedem | have you ever noticed this? https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L2010 | |
| 22:51:34 | mriedem | i bet ^ is wrong now with the cellsv2 local delete case too | |
| 22:51:39 | melwitt | yeah, I was just curious how it happened, mostly to make sure that's the only problem and there isn't some additional problem | |
| 22:52:35 | melwitt | yeah, I have seen that before but it gets glossed over bc soft-delete | |
| 22:53:14 | melwitt | it gets removed in my series (all reserve/commit/rollback removed) | |
| 22:53:41 | mriedem | i noticed it because of this https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/manager.py#L2334 | |
| 22:53:55 | mriedem | the dumb thing is that we create the reservation in the api then just roll it back right away | |
| 22:54:05 | mriedem | we'd be smarter to just not create the reservations in the first place if it's soft delete | |
| 22:54:20 | mriedem | but i'm sure that was fixed as a bolt-on bug fix to the already 30 bolt-ons to the local delete code | |
| 22:54:28 | melwitt | yeah, for real | |
| 22:56:14 | mriedem | ok so maybe rather than me spending time making quotas.commit/rollback fail instead of noop if there are no reservations, because of counting quotas removing all of that in master, i can just work on the functional regression test recreate and the fix since those have to be backported to ocata | |
| 22:56:55 | openstack | Launchpad bug 1675570 in OpenStack Compute (nova) "Volumes are not detached when deleting shelved offloaded servers" [High,In progress] - Assigned to Zhenyu Zheng (zhengzhenyu) | |
| 22:56:55 | mriedem | i wanted to fix this before https://bugs.launchpad.net/nova/+bug/1675570 | |
| 22:58:13 | melwitt | yeah. I wonder if some of that will get consolidated bc of the need to call _local_delete in some cases | |
| 22:58:23 | mriedem | that's my plan | |
| 22:58:34 | mriedem | rather than duplicate/triplicate everything, just re-use _local_delete | |
| 22:58:37 | mriedem | for the latter bug | |
| 22:59:15 | melwitt | _local_delete is needed if the instance had ever been scheduled. which you can't tell from only instance.host bc of shelved offloaded I guess | |
| 23:00:04 | melwitt | the other case (the 5-time-reverted patch I have up) is an instance in error state. it could have volumes attached even though it has instance.host = None | |
| 23:00:34 | melwitt | because there's code in compute that resets instance.host to None when setting to ERROR state | |
| 23:00:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Make quotas.commit/rollback strict https://review.openstack.org/452346 | |
| 23:25:15 | nrezinorn | How new is the server.create_image snapshot function in nova client/api? I am seeing a problem where randomly the snapshot fails, and the VM stays powered off. | |
| 23:28:11 | mriedem | nrezinorn: probably extremely old | |
| 23:28:14 | mriedem | git blame will tell you | |
| 23:28:30 | mriedem | the client side code shouldn't be causing random issues, you're likely hitting something on the server side | |
| 23:28:47 | nrezinorn | its definately stuff server side, as you can see the errors here | |
| 23:28:55 | nrezinorn | https://gist.github.com/Nrezinorn/decc29d0f142eee34c693e7220090ad9 | |
| 23:29:29 | mriedem | qemu is blowing up | |
| 23:29:30 | nrezinorn | ^in that i took a snapshot fine on that VM ~ 20-30 prior and it worked fine | |
| 23:29:50 | nrezinorn | it doesnt always blow up. | |
| 23:31:01 | nrezinorn | i will look through more RH bugreports for qemu , is there anything i can enable on the hupervisor to get more info than the error message from nova-compute? | |
| 23:31:12 | mriedem | libvirt debug logging | |
| 23:31:20 | mriedem | sec | |
| 23:31:43 | mriedem | this is how we setup libvirt debug logging for our CI runs https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/functions-libvirt#L104 | |
| 23:32:01 | mriedem | what version of qemu/libvirt? | |
| 23:33:14 | nrezinorn | libvirt-2.0.0-10.el7_3.4.x86_64 qemu-kvm-1.5.3-126.el7_3.3.x86_64 | |
| 23:34:24 | mriedem | hmm, idk then, we're using what's in ubuntu 16.04 right now which is libvirt 1.3.1 | |
| 23:34:41 | nrezinorn | sorry this is on centos ;) | |
| 23:34:56 | nrezinorn | and i already looked to see if there were updated packages for both qemu and libvirt | |
| 23:35:09 | mriedem | that qemu looks pretty old actually | |
| 23:35:12 | mriedem | https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix | |
| 23:35:12 | nrezinorn | i will attempt the debug stuff you link on monday to see what is really going on. | |
| 23:35:26 | nrezinorn | pretty old == CentOS7 pretty old ;) | |
| 23:35:32 | mriedem | 7.3 2016-11-03 2.0.0 2.6.0 | |
| 23:35:36 | mriedem | is it centos 7.3? | |
| 23:35:39 | nrezinorn | yes | |
| 23:35:40 | mriedem | oh 7.0 | |
| 23:35:51 | mriedem | if it's 7.3, ^ says you should have qemu 2.6.0 | |
| 23:35:56 | mriedem | not 1.5.3 | |
| 23:36:21 | nrezinorn | hmmm | |
| 23:36:23 | nrezinorn | ok thanks | |
| 23:36:26 | mriedem | np | |
| 23:36:42 | nrezinorn | i will look into why the packages are not lining up :) | |
| 23:38:32 | nrezinorn | hmm, wonder where that 2.6.0 rpm lives...on my machine im on now its 7.3 and i only see 1.5.3 like above | |
| 23:44:07 | mriedem | melwitt: ah writing a regression test for this commit/rollback thing shows me yet another problem :) | |
| 23:44:12 | mriedem | it's a bountiful harvest of bugs | |
| 23:44:21 | melwitt | :( | |
| 23:45:10 | mriedem | nrezinorn: i'm not sure which service channels are used to build that list, or if it's just stock (i assume it's supposed to be stock) | |
| 23:45:34 | mriedem | you could ask sgordon but i think it's too late in the day for his tz | |
| 23:46:41 | mriedem | melwitt: right here https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1891 | |
| 23:46:46 | mriedem | we don't return | |
| 23:46:57 | mriedem | but we have an instance, and continue going down the old path | |
| 23:47:41 | melwitt | wah wah | |
| 23:47:55 | mriedem | maybe that's ok, idk | |
| 23:48:24 | mriedem | https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1836-L1906 was all really supposed to be for transition i think | |
| 23:48:54 | mriedem | still have the cell0 case though | |
| 23:49:05 | melwitt | it looks like it should return in that case | |
| 23:49:09 | mriedem | where the quotas are in the main db and the instance is in the cell0 db | |
| 23:49:13 | mriedem | yeah i'm adding that | |
| 23:51:40 | dtp | mriedem is the task of adding a uuid to service model still mine? | |
| 23:51:57 | nrezinorn | mriedem: im going to updgrade to those 2.6.0 RPMs Monday in our testing env - hoping this doesnt bite me though: https://www.redhat.com/archives/rdo-list/2016-December/msg00028.html | |
| 23:52:13 | mriedem | dtp: it is | |
| 23:52:24 | nrezinorn | i'll know more next week, and will come back if i need another drink from the well of knowledge ;) | |
| 23:52:24 | dtp | cool, planning to start it monday | |
| 23:52:40 | mriedem | nrezinorn: you could ask dmisard in #openstack-operators | |
| 23:52:48 | mriedem | it's a shallow well with me | |
| 23:52:55 | mriedem | but dmisard is around other channels | |
| 23:53:12 | mriedem | dmsimard i mean | |
| 23:53:12 | nrezinorn | ill do so monday! 5pm on a friday for me :) | |
| 23:53:18 | mriedem | 7pm for me | |
| 23:53:20 | mriedem | good luck | |
| 23:59:28 | openstack | bug 1678326 in OpenStack Compute (nova) "Quota is decremented during instance delete in cell0 even if the instance destroy fails" [High,Triaged] https://launchpad.net/bugs/1678326 - Assigned to Matt Riedemann (mriedem) | |
| 23:59:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for quota decrement bug 1678326 https://review.openstack.org/452351 | |
| 23:59:31 | mriedem | melwitt: here is the regression test ^ i haven't run unit tests on the bug fix in nova.compute.api | |
| 23:59:33 | mriedem | will do that later | |
| 23:59:38 | mriedem | i'm in trouble for being late to dinner | |
| 23:59:57 | melwitt | k, I'll check it out | |
| #openstack-nova - 2017-04-01 | |||
| 00:57:45 | openstack | bug 1678326 in OpenStack Compute (nova) "Quota is decremented during instance delete in cell0 even if the instance destroy fails" [High,In progress] https://launchpad.net/bugs/1678326 - Assigned to Matt Riedemann (mriedem) | |
| 00:57:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for quota decrement bug 1678326 https://review.openstack.org/452351 | |
| 01:30:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Commit usage decrement after destroying instance https://review.openstack.org/452359 | |
| 01:30:21 | openstack | bug 1678326 in OpenStack Compute (nova) "Quota is decremented during instance delete in cell0 even if the instance destroy fails" [High,In progress] https://launchpad.net/bugs/1678326 - Assigned to Matt Riedemann (mriedem) | |
| 01:30:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for quota decrement bug 1678326 https://review.openstack.org/452351 | |