| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 14:56:34 | mriedem | knikolla_phone: instances.deleted = instances.id | |
| 14:56:36 | mriedem | if it's deleted | |
| 14:56:53 | jaypipes | knikolla_phone: but honestly, you shouldn't be setting stuff manually in the DB. Dragons be there. | |
| 14:56:55 | mriedem | dansmith: but i'm wondering if that's too heavy handed, doing the service version check in the api | |
| 14:57:03 | dansmith | mriedem: yeah, I mean, it's a warning and the assumption was that we'd add events over time, so the warning isn't terrible | |
| 14:57:17 | dansmith | mriedem: just depends on whether or not we have different behavior before/after | |
| 14:57:32 | dansmith | if it's just an optimization that will continue to work but slower, then no need to be heavy-handed I think | |
| 14:57:32 | mriedem | my thinking was as a user i do this volume resize thing, and it all says it's fine, | |
| 14:57:46 | mriedem | but the block device in my guest isn't resized to the new size, so i report a support ticket | |
| 14:58:17 | mriedem | the admin has to go figure out through the logs that the event was basically ignored on the ocata compute because it didn't recognize it | |
| 14:58:46 | knikolla_phone | jaypipes: i like dragons. | |
| 14:58:50 | dansmith | ah, yeah, but.. they triggered the resize from the cinder side right? | |
| 14:58:53 | mriedem | we also don't have any policy check for this to effectively disable it per event until your computes are all upgraded | |
| 14:58:58 | mriedem | dansmith: yes | |
| 14:58:58 | dansmith | mriedem: blocking the event from cinder isn't going to tell the user anything right? | |
| 14:59:06 | mriedem | but cinder is pike and the instance is runnign on an ocata compute | |
| 14:59:07 | knikolla_phone | We were left with an inconsistent db after upgrade. Some instances failed to create and delete. But everything else was fine. | |
| 14:59:33 | mriedem | dansmith: i was thinking that if we could detect in the api that this isn't going to work based on service version, then we return a 400 to cinder and cinder rolls back the volume resize | |
| 14:59:37 | mriedem | which is the indication to the user | |
| 14:59:45 | dansmith | mriedem: that's super expensive | |
| 15:00:00 | dansmith | mriedem: and.. can you even roll it back? often times block device resizes are one-way | |
| 15:00:06 | mriedem | alternatively, if cinder has a policy rule for extending an attached volume, then the admin can disable this from the cinder side until they know they can handle it | |
| 15:00:25 | mriedem | mgagne: does the cinder side of this have a unique policy rule for extending attached volumes? | |
| 15:00:34 | mriedem | or is it just the same as os-extend for 'available' non-attached volumes? | |
| 15:01:16 | jaypipes | knikolla_phone: here's a dragon for you: http://bit.ly/2sLVKtR | |
| 15:01:28 | mriedem | mgagne: because if i were a xen or vmware or hyperv shop, and this just will never work, i'd want to disable the cinder api behavior globally | |
| 15:02:28 | knikolla_phone | jaypipes: the nova-manage project quota_refresh_usage command didn't do anything. | |
| 15:02:56 | knikolla_phone | Newton -> ocata was the only problematic upgrade so far for us. :( | |
| 15:04:29 | jaypipes | knikolla_phone: other than manually going through your DB, I'm not entirely sure I can help you on this channel. There's some mismatch somewhere between the project_quota_usages or quota_usages table and the "true" resource tables like instances, instance_extra, pci_devices, etc | |
| 15:04:54 | jaypipes | knikolla_phone: is this isolated to a particular tenant? or general? | |
| 15:06:01 | knikolla_phone | jaypipes: no, that's okay, you've helped more than enough. This is only limited to one tenant. | |
| 15:06:27 | jaypipes | knikolla_phone: and only for their instances quota? | |
| 15:07:32 | knikolla_phone | jaypipes: whatever resources two of the instances we manually deleted used (instances, vcpu, ram, etc) | |
| 15:08:01 | jaypipes | hmm | |
| 15:08:37 | mriedem | dansmith: ok, so i think policy blocker on the cinder side is key, i've put that in the cinder change https://review.openstack.org/#/c/454287/ | |
| 15:08:42 | jaypipes | knikolla_phone: but does the instances usage show correct? meaning, does it show the tenant using instances that were deleted? | |
| 15:08:50 | dansmith | mriedem: ack | |
| 15:08:50 | mriedem | dansmith: and then we just report the shit out of when it fails on the nova compute side in pike | |
| 15:08:58 | mriedem | using action events and faults | |
| 15:11:15 | knikolla_phone | jaypipes: yes. The usage is correct to what it would be if those two instances weren't deleted (which is weird because we manually deleted five of them). Quota_usages table reflects that amount, and updating it to random values and then using quota_refresh_usage command, it preserves the random amount. | |
| 15:13:10 | jaypipes | knikolla_phone: what's in the reservations table for that project? | |
| 15:21:36 | knikolla_phone | jaypipes: what does uuid in that table (reservations) map to? Is it an instance uuid? | |
| 15:21:56 | jaypipes | knikolla_phone: yep | |
| 15:23:18 | knikolla_phone | jaypipes: okay, i see a deleted row. Going to try setting that too. | |
| 15:23:32 | knikolla_phone | Thanks for all your help. | |
| 15:23:36 | mnaser | i've been doing some digging around and i cannot seem to find where root_device_name gets assigned for vms.. can someone give a hint :( | |
| 15:23:42 | jaypipes | knikolla_phone: hey, no prob :) | |
| 15:23:53 | mnaser | i've discovered a bug with when using virtio-scsi with nova | |
| 15:24:06 | jaypipes | mnaser: depends on the hypervisor. | |
| 15:24:13 | jaypipes | mnaser: you talking about libvirt? | |
| 15:24:15 | mnaser | root_device_name gets set to /dev/vda .. but when attaching cinder volumes, it tries to attach them at /dev/sdX, and tries to assign the same pci address | |
| 15:24:16 | mnaser | yep | |
| 15:24:38 | jaypipes | mnaser: hmm... | |
| 15:24:49 | jaypipes | mnaser: might need to bring in mdbooth on that one. | |
| 15:24:51 | mnaser | i turned off the vm, attached the cinder block device and then started it | |
| 15:24:55 | jaypipes | mdbooth: Matt, any ideas? | |
| 15:24:57 | mnaser | and it actually generated invalid libvirt xml | |
| 15:25:01 | mnaser | i have a copy of it, let me paste | |
| 15:26:04 | mnaser | well looks like paste.openstack.org isn't loading | |
| 15:26:11 | mriedem | mnaser: _validate_and_build_base_options in nova.compute.api.API | |
| 15:26:26 | mnaser | https://gist.github.com/mnaser/8b000d20590e4343e4d7a73459d85210 | |
| 15:26:31 | mriedem | sets it in the base_options dict which eventually gets set on the instance (really the BuildRequest now) | |
| 15:26:42 | mnaser | notice the address of the first 2 disks | |
| 15:26:44 | mnaser | is the same | |
| 15:26:47 | mnaser | so qemu never boots the vm | |
| 15:27:29 | mnaser | i added a LOG.error(disk_info) in the libvirt tooling | |
| 15:27:30 | mnaser | and this is what came out of it too - https://gist.github.com/mnaser/aedd44b9df018786a144f7c719ec5a13 | |
| 15:27:49 | mnaser | notice how 'dev': 'vda' (which is ephermal) and the 'dev': 'sda' (which is attached cinder block device) | |
| 15:28:11 | mnaser | i am guessing that's where the issue is stemming, the first one should be sda too (because using virtio-scsi with the image hw_ parameters) | |
| 15:28:12 | mriedem | mnaser: https://github.com/openstack/nova/blob/9045a98145f0834446d668e0282ddb1c6f2d136c/nova/compute/api.py#L921 is eventually what gets set in the instance.root_device_name field | |
| 15:29:22 | mnaser | i've confirmed with two different installs that booting an image with hw_scsi_model=virtio-scsi and hw_disk_bus=scsi will work but no cinder volumes will successfully attach: libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Duplicate ID 'drive-scsi0-0-0-0' for drive | |
| 15:29:30 | mdbooth | mnaser: what's the symptom of the weird xml? | |
| 15:29:35 | mdbooth | Does it barf? | |
| 15:29:50 | mnaser | so, if you try to attach it to an already running vm, you get the libvirt error above (duplicate id) | |
| 15:30:19 | mnaser | if you turn the vm off, attach (really a no-op if its offline), start it, it'll generate that invalid xml and fail | |
| 15:30:32 | mnaser | so im not sure if libvirt or nova is to blame in putting the duplicate address ther | |
| 15:30:45 | mdbooth | We really shouldn't be looking at root_device_name | |
| 15:31:34 | mnaser | i've just been kinda trying to understand the code and i dont know if i can point the finger at nova or libvirt yet | |
| 15:32:03 | mnaser | but it's obviously just an issue when using virtio-scsi | |
| 15:37:48 | mgagne | mriedem: supports_volume_extend vs supports_extend_volume ? RE https://review.openstack.org/#/c/454322/18/nova/virt/driver.py | |
| 15:38:31 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using targeted cell context when finding services in cells https://review.openstack.org/475830 | |
| 15:38:53 | dansmith | dtantsur: mriedem ^ | |
| 15:39:01 | dtantsur | nice | |
| 15:40:09 | openstackgerrit | Kaitlin Farr proposed openstack/nova master: Remove deprecated keymgr code https://review.openstack.org/439855 | |
| 15:41:04 | mriedem | melwitt: i think you might have gotten delete happy in the tests here https://review.openstack.org/#/c/457862/ | |
| 15:41:33 | mriedem | mgagne: i'm ok with either - i suppose supports_extend_volume to match the driver's method name | |
| 15:41:49 | mgagne | that's my opinion | |
| 15:50:22 | mriedem | mdbooth: i think that root_device_name thing is a legacy bdm v1 thing, but not totally sure | |
| 15:50:44 | mriedem | some kind of ec2 carry over | |
| 15:51:36 | mdbooth | mriedem: I haven't followed the code all the way down, but I suspect we could ignore it | |
| 15:54:36 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/475600 | |
| 15:56:50 | dansmith | jaypipes: quick +W bait: https://review.openstack.org/#/c/475830/ | |
| 15:57:01 | dansmith | jaypipes: fixes the ironic people | |
| 15:57:31 | jaypipes | dansmith: wallaby'd | |
| 15:57:38 | dansmith | thanks | |
| 15:57:41 | dtantsur | thanks all! | |
| 15:57:51 | dansmith | I guess cdent still isn't around today? | |
| 15:58:06 | jaypipes | dansmith: he's on PTO for next two weeksd. | |
| 15:58:11 | dansmith | oh my | |
| 15:58:20 | jaypipes | dansmith: after which he will be a VMWare employee. | |