| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 14:49:40 | jaypipes | but pretty sure knikolla_phone doesn't have yesterday's code deployed ;) | |
| 14:49:45 | dansmith | GASP | |
| 14:50:05 | mriedem | hells bells https://review.openstack.org/#/c/475830/1/nova/compute/api.py | |
| 14:50:11 | dansmith | jaypipes: I know, but it's amusing timing given the flurry of them in the gate atm | |
| 14:50:19 | knikolla_phone | Jaypipes: we're running ocata | |
| 14:50:36 | jaypipes | knikolla_phone: yup, so you need to do the nova-manage quota refresh thing :) | |
| 14:50:41 | jaypipes | dansmith: ack | |
| 14:51:29 | knikolla_phone | Jaypipes: thanks. Will give it a try now. | |
| 14:53:00 | mriedem | mgagne: sorry it took awhile, but i finally got another thorough review on https://review.openstack.org/#/c/454322/ | |
| 14:53:10 | mriedem | mgagne: lots of edge cases in there that need to be dealt with | |
| 14:53:23 | mriedem | dansmith: so, | |
| 14:53:34 | mgagne | ok, will read and address them asap | |
| 14:53:37 | mriedem | something i've been meaning to pester other cores about for their opinions on https://review.openstack.org/#/c/454322/ | |
| 14:53:55 | mriedem | dansmith: ^ is a new compute driver capability, | |
| 14:54:05 | mriedem | called by cinder when a volume is resized | |
| 14:54:21 | mriedem | if you're doing this on an instance running on an ocata compute, that's going to be ignored | |
| 14:54:30 | mriedem | with no indication to the user/admin | |
| 14:55:06 | knikolla_phone | jaypipes: that didn't work. Still ghost resources. | |
| 14:55:26 | mriedem | dansmith: well, take that back, you'll get a warning in the compute logs on the ocata node for "Received unexpected event %(event)s for instance" | |
| 14:55:31 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform keypair.delete notification https://review.openstack.org/463946 | |
| 14:56:07 | knikolla_phone | I know i asked this once and mriedem answered. But what does the deleted column in the table need to be set for a deleted instance? I can't remember what it was, sorry to ask again :( | |
| 14:56:09 | mriedem | dansmith: i was pondering doing a minimum compute service version check in the api for these types of events, since we know that you need a minimum of pike to support those events, and we could return a 400 to cinder otherwise, | |
| 14:56:16 | mriedem | and they could rollback the volume resize on their side | |
| 14:56:24 | jaypipes | knikolla_phone: to its ID value. | |
| 14:56:34 | mriedem | knikolla_phone: instances.deleted = instances.id | |
| 14:56:36 | mriedem | if it's deleted | |
| 14:56:53 | jaypipes | knikolla_phone: but honestly, you shouldn't be setting stuff manually in the DB. Dragons be there. | |
| 14:56:55 | mriedem | dansmith: but i'm wondering if that's too heavy handed, doing the service version check in the api | |
| 14:57:03 | dansmith | mriedem: yeah, I mean, it's a warning and the assumption was that we'd add events over time, so the warning isn't terrible | |
| 14:57:17 | dansmith | mriedem: just depends on whether or not we have different behavior before/after | |
| 14:57:32 | mriedem | my thinking was as a user i do this volume resize thing, and it all says it's fine, | |
| 14:57:32 | dansmith | if it's just an optimization that will continue to work but slower, then no need to be heavy-handed I think | |
| 14:57:46 | mriedem | but the block device in my guest isn't resized to the new size, so i report a support ticket | |
| 14:58:17 | mriedem | the admin has to go figure out through the logs that the event was basically ignored on the ocata compute because it didn't recognize it | |
| 14:58:46 | knikolla_phone | jaypipes: i like dragons. | |
| 14:58:50 | dansmith | ah, yeah, but.. they triggered the resize from the cinder side right? | |
| 14:58:53 | mriedem | we also don't have any policy check for this to effectively disable it per event until your computes are all upgraded | |
| 14:58:58 | dansmith | mriedem: blocking the event from cinder isn't going to tell the user anything right? | |
| 14:58:58 | mriedem | dansmith: yes | |
| 14:59:06 | mriedem | but cinder is pike and the instance is runnign on an ocata compute | |
| 14:59:07 | knikolla_phone | We were left with an inconsistent db after upgrade. Some instances failed to create and delete. But everything else was fine. | |
| 14:59:33 | mriedem | dansmith: i was thinking that if we could detect in the api that this isn't going to work based on service version, then we return a 400 to cinder and cinder rolls back the volume resize | |
| 14:59:37 | mriedem | which is the indication to the user | |
| 14:59:45 | dansmith | mriedem: that's super expensive | |
| 15:00:00 | dansmith | mriedem: and.. can you even roll it back? often times block device resizes are one-way | |
| 15:00:06 | mriedem | alternatively, if cinder has a policy rule for extending an attached volume, then the admin can disable this from the cinder side until they know they can handle it | |
| 15:00:25 | mriedem | mgagne: does the cinder side of this have a unique policy rule for extending attached volumes? | |
| 15:00:34 | mriedem | or is it just the same as os-extend for 'available' non-attached volumes? | |
| 15:01:16 | jaypipes | knikolla_phone: here's a dragon for you: http://bit.ly/2sLVKtR | |
| 15:01:28 | mriedem | mgagne: because if i were a xen or vmware or hyperv shop, and this just will never work, i'd want to disable the cinder api behavior globally | |
| 15:02:28 | knikolla_phone | jaypipes: the nova-manage project quota_refresh_usage command didn't do anything. | |
| 15:02:56 | knikolla_phone | Newton -> ocata was the only problematic upgrade so far for us. :( | |
| 15:04:29 | jaypipes | knikolla_phone: other than manually going through your DB, I'm not entirely sure I can help you on this channel. There's some mismatch somewhere between the project_quota_usages or quota_usages table and the "true" resource tables like instances, instance_extra, pci_devices, etc | |
| 15:04:54 | jaypipes | knikolla_phone: is this isolated to a particular tenant? or general? | |
| 15:06:01 | knikolla_phone | jaypipes: no, that's okay, you've helped more than enough. This is only limited to one tenant. | |
| 15:06:27 | jaypipes | knikolla_phone: and only for their instances quota? | |
| 15:07:32 | knikolla_phone | jaypipes: whatever resources two of the instances we manually deleted used (instances, vcpu, ram, etc) | |
| 15:08:01 | jaypipes | hmm | |
| 15:08:37 | mriedem | dansmith: ok, so i think policy blocker on the cinder side is key, i've put that in the cinder change https://review.openstack.org/#/c/454287/ | |
| 15:08:42 | jaypipes | knikolla_phone: but does the instances usage show correct? meaning, does it show the tenant using instances that were deleted? | |
| 15:08:50 | mriedem | dansmith: and then we just report the shit out of when it fails on the nova compute side in pike | |
| 15:08:50 | dansmith | mriedem: ack | |
| 15:08:58 | mriedem | using action events and faults | |
| 15:11:15 | knikolla_phone | jaypipes: yes. The usage is correct to what it would be if those two instances weren't deleted (which is weird because we manually deleted five of them). Quota_usages table reflects that amount, and updating it to random values and then using quota_refresh_usage command, it preserves the random amount. | |
| 15:13:10 | jaypipes | knikolla_phone: what's in the reservations table for that project? | |
| 15:21:36 | knikolla_phone | jaypipes: what does uuid in that table (reservations) map to? Is it an instance uuid? | |
| 15:21:56 | jaypipes | knikolla_phone: yep | |
| 15:23:18 | knikolla_phone | jaypipes: okay, i see a deleted row. Going to try setting that too. | |
| 15:23:32 | knikolla_phone | Thanks for all your help. | |
| 15:23:36 | mnaser | i've been doing some digging around and i cannot seem to find where root_device_name gets assigned for vms.. can someone give a hint :( | |
| 15:23:42 | jaypipes | knikolla_phone: hey, no prob :) | |
| 15:23:53 | mnaser | i've discovered a bug with when using virtio-scsi with nova | |
| 15:24:06 | jaypipes | mnaser: depends on the hypervisor. | |
| 15:24:13 | jaypipes | mnaser: you talking about libvirt? | |
| 15:24:15 | mnaser | root_device_name gets set to /dev/vda .. but when attaching cinder volumes, it tries to attach them at /dev/sdX, and tries to assign the same pci address | |
| 15:24:16 | mnaser | yep | |
| 15:24:38 | jaypipes | mnaser: hmm... | |
| 15:24:49 | jaypipes | mnaser: might need to bring in mdbooth on that one. | |
| 15:24:51 | mnaser | i turned off the vm, attached the cinder block device and then started it | |
| 15:24:55 | jaypipes | mdbooth: Matt, any ideas? | |
| 15:24:57 | mnaser | and it actually generated invalid libvirt xml | |
| 15:25:01 | mnaser | i have a copy of it, let me paste | |
| 15:26:04 | mnaser | well looks like paste.openstack.org isn't loading | |
| 15:26:11 | mriedem | mnaser: _validate_and_build_base_options in nova.compute.api.API | |
| 15:26:26 | mnaser | https://gist.github.com/mnaser/8b000d20590e4343e4d7a73459d85210 | |
| 15:26:31 | mriedem | sets it in the base_options dict which eventually gets set on the instance (really the BuildRequest now) | |
| 15:26:42 | mnaser | notice the address of the first 2 disks | |
| 15:26:44 | mnaser | is the same | |
| 15:26:47 | mnaser | so qemu never boots the vm | |
| 15:27:29 | mnaser | i added a LOG.error(disk_info) in the libvirt tooling | |
| 15:27:30 | mnaser | and this is what came out of it too - https://gist.github.com/mnaser/aedd44b9df018786a144f7c719ec5a13 | |
| 15:27:49 | mnaser | notice how 'dev': 'vda' (which is ephermal) and the 'dev': 'sda' (which is attached cinder block device) | |
| 15:28:11 | mnaser | i am guessing that's where the issue is stemming, the first one should be sda too (because using virtio-scsi with the image hw_ parameters) | |
| 15:28:12 | mriedem | mnaser: https://github.com/openstack/nova/blob/9045a98145f0834446d668e0282ddb1c6f2d136c/nova/compute/api.py#L921 is eventually what gets set in the instance.root_device_name field | |
| 15:29:22 | mnaser | i've confirmed with two different installs that booting an image with hw_scsi_model=virtio-scsi and hw_disk_bus=scsi will work but no cinder volumes will successfully attach: libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Duplicate ID 'drive-scsi0-0-0-0' for drive | |
| 15:29:30 | mdbooth | mnaser: what's the symptom of the weird xml? | |
| 15:29:35 | mdbooth | Does it barf? | |
| 15:29:50 | mnaser | so, if you try to attach it to an already running vm, you get the libvirt error above (duplicate id) | |
| 15:30:19 | mnaser | if you turn the vm off, attach (really a no-op if its offline), start it, it'll generate that invalid xml and fail | |
| 15:30:32 | mnaser | so im not sure if libvirt or nova is to blame in putting the duplicate address ther | |