| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: explain payload inheritance in notification devref https://review.openstack.org/453667 | |
| 14:36:30 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:36:48 | knikolla_phone | How does nova calculate quota usage? After manually setting an instance as deleted in the database 'instances' table the resources still show up as used. | |
| 14:37:08 | dansmith | heh | |
| 14:37:12 | dtantsur | dansmith: https://paste.fedoraproject.org/paste/r26SqCAGrk8dXPlOk-sfjg | |
| 14:39:27 | dansmith | dtantsur: that's kinda hard to make sense of, especially without seeing the code, but.. why the HostMappingNotFound messages interleaved with the iteration of that for loop? | |
| 14:39:51 | dtantsur | dansmith: I think it was a bug in oslo.log | |
| 14:39:57 | dansmith | oh eff me | |
| 14:40:00 | dansmith | dtantsur: I see the bug | |
| 14:41:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using targeted cell context when finding services in cells https://review.openstack.org/475830 | |
| 14:41:30 | dansmith | dtantsur: try that ^ :) | |
| 14:41:37 | dtantsur | thanks! | |
| 14:42:19 | dansmith | dtantsur: it was iterating but not actually looking in the other database :P | |
| 14:42:27 | dtantsur | I see | |
| 14:46:33 | dansmith | dtantsur: if this fixes you, then: sorry for wasting your time | |
| 14:47:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/475600 | |
| 14:48:04 | dtantsur | dansmith: w00t, it works with this patch :) thanks! | |
| 14:48:24 | dansmith | dtantsur: cool, I shall write a test and then try to get that in asap | |
| 14:48:31 | dtantsur | great | |
| 14:48:39 | jaypipes | knikolla_phone: you need to run the nova-manage quota refresh command. | |
| 14:48:50 | dansmith | cc: mriedem | |
| 14:49:03 | jaypipes | knikolla_phone: unfortunately, quotas uses different tables than instances to store usage. | |
| 14:49:20 | dansmith | jaypipes: yesterday that was true | |
| 14:49:23 | dansmith | jaypipes: today it may not be | |
| 14:49:25 | jaypipes | wait, what? | |
| 14:49:28 | jaypipes | oh, yeah. | |
| 14:49:40 | jaypipes | but pretty sure knikolla_phone doesn't have yesterday's code deployed ;) | |
| 14:49:45 | dansmith | GASP | |
| 14:50:05 | mriedem | hells bells https://review.openstack.org/#/c/475830/1/nova/compute/api.py | |
| 14:50:11 | dansmith | jaypipes: I know, but it's amusing timing given the flurry of them in the gate atm | |
| 14:50:19 | knikolla_phone | Jaypipes: we're running ocata | |
| 14:50:36 | jaypipes | knikolla_phone: yup, so you need to do the nova-manage quota refresh thing :) | |
| 14:50:41 | jaypipes | dansmith: ack | |
| 14:51:29 | knikolla_phone | Jaypipes: thanks. Will give it a try now. | |
| 14:53:00 | mriedem | mgagne: sorry it took awhile, but i finally got another thorough review on https://review.openstack.org/#/c/454322/ | |
| 14:53:10 | mriedem | mgagne: lots of edge cases in there that need to be dealt with | |
| 14:53:23 | mriedem | dansmith: so, | |
| 14:53:34 | mgagne | ok, will read and address them asap | |
| 14:53:37 | mriedem | something i've been meaning to pester other cores about for their opinions on https://review.openstack.org/#/c/454322/ | |
| 14:53:55 | mriedem | dansmith: ^ is a new compute driver capability, | |
| 14:54:05 | mriedem | called by cinder when a volume is resized | |
| 14:54:21 | mriedem | if you're doing this on an instance running on an ocata compute, that's going to be ignored | |
| 14:54:30 | mriedem | with no indication to the user/admin | |
| 14:55:06 | knikolla_phone | jaypipes: that didn't work. Still ghost resources. | |
| 14:55:26 | mriedem | dansmith: well, take that back, you'll get a warning in the compute logs on the ocata node for "Received unexpected event %(event)s for instance" | |
| 14:55:31 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform keypair.delete notification https://review.openstack.org/463946 | |
| 14:56:07 | knikolla_phone | I know i asked this once and mriedem answered. But what does the deleted column in the table need to be set for a deleted instance? I can't remember what it was, sorry to ask again :( | |
| 14:56:09 | mriedem | dansmith: i was pondering doing a minimum compute service version check in the api for these types of events, since we know that you need a minimum of pike to support those events, and we could return a 400 to cinder otherwise, | |
| 14:56:16 | mriedem | and they could rollback the volume resize on their side | |
| 14:56:24 | jaypipes | knikolla_phone: to its ID value. | |
| 14:56:34 | mriedem | knikolla_phone: instances.deleted = instances.id | |
| 14:56:36 | mriedem | if it's deleted | |
| 14:56:53 | jaypipes | knikolla_phone: but honestly, you shouldn't be setting stuff manually in the DB. Dragons be there. | |
| 14:56:55 | mriedem | dansmith: but i'm wondering if that's too heavy handed, doing the service version check in the api | |
| 14:57:03 | dansmith | mriedem: yeah, I mean, it's a warning and the assumption was that we'd add events over time, so the warning isn't terrible | |
| 14:57:17 | dansmith | mriedem: just depends on whether or not we have different behavior before/after | |
| 14:57:32 | mriedem | my thinking was as a user i do this volume resize thing, and it all says it's fine, | |
| 14:57:32 | dansmith | if it's just an optimization that will continue to work but slower, then no need to be heavy-handed I think | |
| 14:57:46 | mriedem | but the block device in my guest isn't resized to the new size, so i report a support ticket | |
| 14:58:17 | mriedem | the admin has to go figure out through the logs that the event was basically ignored on the ocata compute because it didn't recognize it | |
| 14:58:46 | knikolla_phone | jaypipes: i like dragons. | |
| 14:58:50 | dansmith | ah, yeah, but.. they triggered the resize from the cinder side right? | |
| 14:58:53 | mriedem | we also don't have any policy check for this to effectively disable it per event until your computes are all upgraded | |
| 14:58:58 | dansmith | mriedem: blocking the event from cinder isn't going to tell the user anything right? | |
| 14:58:58 | mriedem | dansmith: yes | |
| 14:59:06 | mriedem | but cinder is pike and the instance is runnign on an ocata compute | |
| 14:59:07 | knikolla_phone | We were left with an inconsistent db after upgrade. Some instances failed to create and delete. But everything else was fine. | |
| 14:59:33 | mriedem | dansmith: i was thinking that if we could detect in the api that this isn't going to work based on service version, then we return a 400 to cinder and cinder rolls back the volume resize | |
| 14:59:37 | mriedem | which is the indication to the user | |
| 14:59:45 | dansmith | mriedem: that's super expensive | |
| 15:00:00 | dansmith | mriedem: and.. can you even roll it back? often times block device resizes are one-way | |
| 15:00:06 | mriedem | alternatively, if cinder has a policy rule for extending an attached volume, then the admin can disable this from the cinder side until they know they can handle it | |
| 15:00:25 | mriedem | mgagne: does the cinder side of this have a unique policy rule for extending attached volumes? | |
| 15:00:34 | mriedem | or is it just the same as os-extend for 'available' non-attached volumes? | |
| 15:01:16 | jaypipes | knikolla_phone: here's a dragon for you: http://bit.ly/2sLVKtR | |
| 15:01:28 | mriedem | mgagne: because if i were a xen or vmware or hyperv shop, and this just will never work, i'd want to disable the cinder api behavior globally | |
| 15:02:28 | knikolla_phone | jaypipes: the nova-manage project quota_refresh_usage command didn't do anything. | |
| 15:02:56 | knikolla_phone | Newton -> ocata was the only problematic upgrade so far for us. :( | |
| 15:04:29 | jaypipes | knikolla_phone: other than manually going through your DB, I'm not entirely sure I can help you on this channel. There's some mismatch somewhere between the project_quota_usages or quota_usages table and the "true" resource tables like instances, instance_extra, pci_devices, etc | |
| 15:04:54 | jaypipes | knikolla_phone: is this isolated to a particular tenant? or general? | |
| 15:06:01 | knikolla_phone | jaypipes: no, that's okay, you've helped more than enough. This is only limited to one tenant. | |
| 15:06:27 | jaypipes | knikolla_phone: and only for their instances quota? | |
| 15:07:32 | knikolla_phone | jaypipes: whatever resources two of the instances we manually deleted used (instances, vcpu, ram, etc) | |
| 15:08:01 | jaypipes | hmm | |
| 15:08:37 | mriedem | dansmith: ok, so i think policy blocker on the cinder side is key, i've put that in the cinder change https://review.openstack.org/#/c/454287/ | |
| 15:08:42 | jaypipes | knikolla_phone: but does the instances usage show correct? meaning, does it show the tenant using instances that were deleted? | |
| 15:08:50 | mriedem | dansmith: and then we just report the shit out of when it fails on the nova compute side in pike | |
| 15:08:50 | dansmith | mriedem: ack | |
| 15:08:58 | mriedem | using action events and faults | |
| 15:11:15 | knikolla_phone | jaypipes: yes. The usage is correct to what it would be if those two instances weren't deleted (which is weird because we manually deleted five of them). Quota_usages table reflects that amount, and updating it to random values and then using quota_refresh_usage command, it preserves the random amount. | |
| 15:13:10 | jaypipes | knikolla_phone: what's in the reservations table for that project? | |
| 15:21:36 | knikolla_phone | jaypipes: what does uuid in that table (reservations) map to? Is it an instance uuid? | |
| 15:21:56 | jaypipes | knikolla_phone: yep | |
| 15:23:18 | knikolla_phone | jaypipes: okay, i see a deleted row. Going to try setting that too. | |
| 15:23:32 | knikolla_phone | Thanks for all your help. | |
| 15:23:36 | mnaser | i've been doing some digging around and i cannot seem to find where root_device_name gets assigned for vms.. can someone give a hint :( | |
| 15:23:42 | jaypipes | knikolla_phone: hey, no prob :) | |
| 15:23:53 | mnaser | i've discovered a bug with when using virtio-scsi with nova | |
| 15:24:06 | jaypipes | mnaser: depends on the hypervisor. | |
| 15:24:13 | jaypipes | mnaser: you talking about libvirt? | |