| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 19:49:52 | openstackgerrit | Jon Bernard proposed openstack/nova master: Add tempest-dsvm-ceph-rc https://review.openstack.org/456292 | |
| 19:53:39 | openstackgerrit | Jon Bernard proposed openstack/nova master: Add tempest-dsvm-ceph-rc https://review.openstack.org/456292 | |
| 19:55:38 | Laszlo_ | hello | |
| 19:56:01 | Laszlo_ | I've run into an interesting situation with Newton: | |
| 19:56:20 | Laszlo_ | I have a user user1 in project1 | |
| 19:56:28 | Laszlo_ | the user is creating a VM | |
| 19:56:50 | Laszlo_ | then the admin user in the admin project is creating a volume | |
| 19:57:59 | Laszlo_ | next the admin user is attaching the volume using the ID of the VM (server) created by the user and the ID of the voulme created by admin | |
| 19:58:38 | Laszlo_ | all fine, the volume is visible inside the server. | |
| 19:59:05 | mriedem | user tries to detach the volume or delete the instance and it blows up because the user doesn't have access to GET the volume | |
| 19:59:08 | mriedem | by design | |
| 19:59:27 | Laszlo_ | next the user is deleting the server (actually is deleting the entire stack that has created the server) | |
| 19:59:52 | Laszlo_ | openstack stack delete lab2 | |
| 20:00:03 | Laszlo_ | no errors so far | |
| 20:00:35 | Laszlo_ | but when I'm looking at the volume created by the admin user I can still see it as being attached .... | |
| 20:00:48 | mriedem | because nova failed to detach it | |
| 20:00:55 | mriedem | because the instance user doesn't have access to the volume | |
| 20:01:16 | Laszlo_ | I see ... | |
| 20:01:24 | mriedem | fails here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2267 | |
| 20:01:39 | Laszlo_ | can I run the detach as admin? (now the VM is already gone .... ) | |
| 20:01:40 | mriedem | you should see one of those warnings in the logs | |
| 20:01:54 | mriedem | you'd have to detach the volume as admin before deleting the server | |
| 20:02:00 | mriedem | you can't detach it after the server is gone | |
| 20:02:11 | mriedem | otherwise you need to force a reset of the volume state on the cinder side | |
| 20:02:32 | Laszlo_ | is there a way out from this situation? | |
| 20:03:22 | mriedem | Laszlo_: i think you have to use the os-reset_status API on the cinder side in that case | |
| 20:04:06 | mriedem | https://developer.openstack.org/api-ref/block-storage/v2/index.html#reset-volume-statuses | |
| 20:04:23 | Laszlo_ | mriedem: OK. I will chack it. I have not used this before | |
| 20:04:30 | mriedem | it's really a backdoor hack | |
| 20:04:35 | mriedem | when something goes wrong | |
| 20:04:47 | mriedem | your root problem is that you have the admin creating and attaching a volume to a server it does not own | |
| 20:05:13 | mriedem | so when the server is deleted, the owner of the server does not have access to detach the volume | |
| 20:05:19 | mriedem | b/c it can't get access to the volume | |
| 20:05:22 | mriedem | since the admin owns it | |
| 20:05:37 | mriedem | so either, use a single tenant, or make sure your tooling has the admin detach the volume before the server is deleted | |
| 20:07:41 | Laszlo_ | mriedem: I understood the problem. currently the issue is not big because I was just playing with a newton lab install | |
| 20:08:24 | mriedem | whew | |
| 20:08:29 | Laszlo_ | so it's a good chance to learn about how to solve the issue if I will have it in real life situation ... :) | |
| 20:21:39 | openstackgerrit | Vladik Romanovsky proposed openstack/nova master: Add spice-native type https://review.openstack.org/454940 | |
| 20:29:17 | smcginnis | Laszlo_: In case you don't want to make API calls directly, here's the client info: https://docs.openstack.org/cli-reference/cinder.html | |
| 20:32:19 | Laszlo_ | smcginnis: thank you! that makes our life much easyer ... :) | |
| 20:32:57 | smcginnis | ;) | |
| 20:33:26 | Laszlo_ | so the right command would be: cinder reset-state --state available --attach-status detached right? | |
| 20:33:52 | smcginnis | Laszlo_: Not having done it lately and not having access to tests right now... yes, I believe so. :) | |
| 20:34:09 | Laszlo_ | one sec .. I'll tell you ... :) | |
| 20:35:39 | Laszlo_ | it worked! of course I had to specify the vol_id also ... | |
| 20:35:48 | Laszlo_ | thank you! | |
| 20:44:11 | mriedem | gmann: did you ever start this? https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests | |
| 20:58:07 | mriedem | dansmith: jaypipes: so i think i'm going to push a change to deprecate the TypeAffinityFilter, there wasn't much reply in the mailing list on it, but as noted there it's broken by design if you ever change your flavors | |
| 20:58:39 | mriedem | http://lists.openstack.org/pipermail/openstack-operators/2017-April/013131.html | |
| 21:01:02 | jaypipes | mriedem: cool with me, but then again, I'm pro-removal-of-most-things. ;) | |
| 21:05:32 | dansmith | mriedem: yes, me as well | |
| 21:22:44 | melwitt | anyone else seen random failures of nova.tests.unit.virt.ironic.test_driver.IronicDriverConsoleTestCase.test__get_node_console_with_reset_wait_timeout locally? | |
| 21:23:32 | dansmith | not locally, but have seen it twice in the last few hours in infra | |
| 21:24:08 | melwitt | something is afoot, it's not just me | |
| 21:24:45 | melwitt | :) | |
| 21:28:35 | mriedem | jaypipes: dansmith: btw, why do we still have ram/disk filters in enabled_filters? | |
| 21:28:42 | mriedem | placement replaces those now | |
| 21:28:46 | mriedem | yeah? | |
| 21:28:49 | dansmith | yeah, might should remove them | |
| 21:28:56 | dansmith | I hear jaypipes loves ramfilter though, so maybe he wants to keep it | |
| 21:29:04 | mriedem | EXCEPT | |
| 21:29:11 | mriedem | ramfilter checks host_state.free_ram_mb | |
| 21:29:19 | mriedem | which has that overhead thing in it from the virt drivers | |
| 21:29:22 | mriedem | and placement doesn't | |
| 21:30:47 | mriedem | so maybe we can't remove it until that bug is fixed | |
| 21:30:49 | dansmith | weren't we going to stop writing the RT data in pike though? | |
| 21:30:52 | mriedem | well, remove it from default | |
| 21:31:09 | mriedem | writing what rt data? | |
| 21:32:29 | dansmith | all the existing stuff we're duplicating in placement now | |
| 21:32:38 | mriedem | melwitt: https://github.com/openstack/nova/commit/213f7120c483a37317091b7478dfb99495619ce4 ? | |
| 21:32:41 | mriedem | for that ironic thing | |
| 21:32:54 | mriedem | dansmith: oh writing to the compute nodes table? | |
| 21:33:17 | dansmith | yeah, all the stats we compile in the RT from the instances, available resources, etc | |
| 21:33:30 | mriedem | i don't remember that specifically | |
| 21:33:32 | mriedem | for pike | |
| 21:33:41 | mriedem | i remember deprecating the ram/disk/cpu filters | |
| 21:33:49 | mriedem | and sylvain had the nova-status thing for it kind of | |
| 21:33:50 | dansmith | unless we're going to keep it for the ramfilter, but that would suck | |
| 21:33:50 | dansmith | because right now we're keeping track of the data twice, which we needed for the upgrade, but at pike we don't, | |
| 21:34:11 | mriedem | https://review.openstack.org/#/c/427200/ | |
| 21:34:57 | mriedem | yeah this was the thing where the filters are enabled (which ram and disk are by default), and they had allocation ratios set | |
| 21:35:05 | mriedem | because you didn't set reserved=9999 on your provider | |
| 21:36:17 | mriedem | anyway, seems we should take those out of the default enabled filters | |
| 21:36:29 | mriedem | but probably need to fix that overhead bug first | |
| 21:58:14 | openstackgerrit | Steven Webster proposed openstack/nova master: Fix mitaka online migration for PCI devices https://review.openstack.org/456397 | |
| 21:58:41 | openstackgerrit | Steven Webster proposed openstack/nova master: Fix port update exception when unshelving an instance with PCI devices https://review.openstack.org/453938 | |
| 22:17:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate TypeAffinityFilter https://review.openstack.org/457812 | |
| 22:23:00 | mriedem | nova.tests.unit.virt.ironic.test_driver.IronicDriverConsoleTestCase.test__get_node_console_with_reset_wait_timeout is definitely failing due to something changing with the enforce_type stuff in https://review.openstack.org/#/c/457188/ <- gcb | |
| 22:23:25 | mriedem | oh because it changed from a 0.1 to a 1 | |
| 22:24:36 | mriedem | and _CONSOLE_STATE_CHECKING_INTERVAL is set to 0.05 | |
| 22:25:43 | mriedem | jitter=0.5 in the driver | |
| 22:33:39 | openstack | Launchpad bug 1683953 in OpenStack Compute (nova) "IronicDriverConsoleTestCase.test__get_node_console_with_reset_wait_timeout intermittently fails since 4/18" [Medium,Confirmed] | |
| 22:33:39 | mriedem | https://bugs.launchpad.net/nova/+bug/1683953 | |
| 23:34:10 | gmann | mriedem: not yet, i am planning after summit and once we have extensions removal in good progress and more people can join hands there | |
| 23:47:55 | mriedem | ok | |
| #openstack-nova - 2017-04-19 | |||
| 00:42:15 | mriedem | dansmith: the claims in scheduler spec is close https://review.openstack.org/#/c/437424/ - i've posted some ideas on how to handle the issue with the overhead estimates coming from the virt driver which are used in the claim, and which we won't have in the controller layer | |
| 01:28:18 | openstackgerrit | STEW TY proposed openstack/nova master: Transform instance.unrescue notifications https://review.openstack.org/388275 | |
| 02:18:15 | Kevin_Zheng | mriedem: Hi, I replied on https://bugs.launchpad.net/nova/+bug/1645175 | |
| 02:18:16 | openstack | Launchpad bug 1645175 in OpenStack Compute (nova) "Neutron port got deleted when attach interface failed" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |