| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 14:27:41 | sdague | mriedem: ok, I went up until the first jenkins failing patch | |
| 14:27:54 | mriedem | sdague: that's the way it's always been | |
| 14:27:56 | sdague | and they look good | |
| 14:28:11 | sdague | mriedem: yeh, I didn't realize, but the follow on patch touched a line where it was more obvious | |
| 14:29:09 | mriedem | thanks for hitting those | |
| 14:29:09 | dtantsur | dansmith: sorry for confusing. the cells are ok. I'm just getting the same host out of both.. | |
| 14:29:59 | dansmith | dtantsur: okay | |
| 14:31:13 | dtantsur | but 'services' table is empty on nova_cell0, I've just double-checked >_< | |
| 14:31:44 | sdague | mriedem: my guess is that got caught up in the 6 way octopus merge limits with git | |
| 14:32:01 | sdague | mriedem: so honestly rebasing everything once the base set lands is probably in order anyway | |
| 14:32:36 | dansmith | dtantsur: and just to be clear, the api_connection is pointed at nova_api and not nova_placement right? | |
| 14:32:46 | dtantsur | lemme check too | |
| 14:33:09 | sdague | mriedem: my only wish in those patches (which is long past as it landed in march) is that the config option was strict_quotas instead of recheck_quotas, just from a nice name perspective | |
| 14:34:31 | dtantsur | dansmith: [api_database]connection points to nova_api | |
| 14:34:59 | dansmith | dtantsur: okay so in that for loop, is it finding/setting | |
| 14:35:06 | dansmith | service to anything on the cell0 pass? | |
| 14:35:30 | dtantsur | dansmith: yep. lemme show you the output of my added logging | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add snapshot id to the snapshot notifications https://review.openstack.org/453077 | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: explain payload inheritance in notification devref https://review.openstack.org/453667 | |
| 14:36:30 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:36:48 | knikolla_phone | How does nova calculate quota usage? After manually setting an instance as deleted in the database 'instances' table the resources still show up as used. | |
| 14:37:08 | dansmith | heh | |
| 14:37:12 | dtantsur | dansmith: https://paste.fedoraproject.org/paste/r26SqCAGrk8dXPlOk-sfjg | |
| 14:39:27 | dansmith | dtantsur: that's kinda hard to make sense of, especially without seeing the code, but.. why the HostMappingNotFound messages interleaved with the iteration of that for loop? | |
| 14:39:51 | dtantsur | dansmith: I think it was a bug in oslo.log | |
| 14:39:57 | dansmith | oh eff me | |
| 14:40:00 | dansmith | dtantsur: I see the bug | |
| 14:41:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using targeted cell context when finding services in cells https://review.openstack.org/475830 | |
| 14:41:30 | dansmith | dtantsur: try that ^ :) | |
| 14:41:37 | dtantsur | thanks! | |
| 14:42:19 | dansmith | dtantsur: it was iterating but not actually looking in the other database :P | |
| 14:42:27 | dtantsur | I see | |
| 14:46:33 | dansmith | dtantsur: if this fixes you, then: sorry for wasting your time | |
| 14:47:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/475600 | |
| 14:48:04 | dtantsur | dansmith: w00t, it works with this patch :) thanks! | |
| 14:48:24 | dansmith | dtantsur: cool, I shall write a test and then try to get that in asap | |
| 14:48:31 | dtantsur | great | |
| 14:48:39 | jaypipes | knikolla_phone: you need to run the nova-manage quota refresh command. | |
| 14:48:50 | dansmith | cc: mriedem | |
| 14:49:03 | jaypipes | knikolla_phone: unfortunately, quotas uses different tables than instances to store usage. | |
| 14:49:20 | dansmith | jaypipes: yesterday that was true | |
| 14:49:23 | dansmith | jaypipes: today it may not be | |
| 14:49:25 | jaypipes | wait, what? | |
| 14:49:28 | jaypipes | oh, yeah. | |
| 14:49:40 | jaypipes | but pretty sure knikolla_phone doesn't have yesterday's code deployed ;) | |
| 14:49:45 | dansmith | GASP | |
| 14:50:05 | mriedem | hells bells https://review.openstack.org/#/c/475830/1/nova/compute/api.py | |
| 14:50:11 | dansmith | jaypipes: I know, but it's amusing timing given the flurry of them in the gate atm | |
| 14:50:19 | knikolla_phone | Jaypipes: we're running ocata | |
| 14:50:36 | jaypipes | knikolla_phone: yup, so you need to do the nova-manage quota refresh thing :) | |
| 14:50:41 | jaypipes | dansmith: ack | |
| 14:51:29 | knikolla_phone | Jaypipes: thanks. Will give it a try now. | |
| 14:53:00 | mriedem | mgagne: sorry it took awhile, but i finally got another thorough review on https://review.openstack.org/#/c/454322/ | |
| 14:53:10 | mriedem | mgagne: lots of edge cases in there that need to be dealt with | |
| 14:53:23 | mriedem | dansmith: so, | |
| 14:53:34 | mgagne | ok, will read and address them asap | |
| 14:53:37 | mriedem | something i've been meaning to pester other cores about for their opinions on https://review.openstack.org/#/c/454322/ | |
| 14:53:55 | mriedem | dansmith: ^ is a new compute driver capability, | |
| 14:54:05 | mriedem | called by cinder when a volume is resized | |
| 14:54:21 | mriedem | if you're doing this on an instance running on an ocata compute, that's going to be ignored | |
| 14:54:30 | mriedem | with no indication to the user/admin | |
| 14:55:06 | knikolla_phone | jaypipes: that didn't work. Still ghost resources. | |
| 14:55:26 | mriedem | dansmith: well, take that back, you'll get a warning in the compute logs on the ocata node for "Received unexpected event %(event)s for instance" | |
| 14:55:31 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform keypair.delete notification https://review.openstack.org/463946 | |
| 14:56:07 | knikolla_phone | I know i asked this once and mriedem answered. But what does the deleted column in the table need to be set for a deleted instance? I can't remember what it was, sorry to ask again :( | |
| 14:56:09 | mriedem | dansmith: i was pondering doing a minimum compute service version check in the api for these types of events, since we know that you need a minimum of pike to support those events, and we could return a 400 to cinder otherwise, | |
| 14:56:16 | mriedem | and they could rollback the volume resize on their side | |
| 14:56:24 | jaypipes | knikolla_phone: to its ID value. | |
| 14:56:34 | mriedem | knikolla_phone: instances.deleted = instances.id | |
| 14:56:36 | mriedem | if it's deleted | |
| 14:56:53 | jaypipes | knikolla_phone: but honestly, you shouldn't be setting stuff manually in the DB. Dragons be there. | |
| 14:56:55 | mriedem | dansmith: but i'm wondering if that's too heavy handed, doing the service version check in the api | |
| 14:57:03 | dansmith | mriedem: yeah, I mean, it's a warning and the assumption was that we'd add events over time, so the warning isn't terrible | |
| 14:57:17 | dansmith | mriedem: just depends on whether or not we have different behavior before/after | |
| 14:57:32 | mriedem | my thinking was as a user i do this volume resize thing, and it all says it's fine, | |
| 14:57:32 | dansmith | if it's just an optimization that will continue to work but slower, then no need to be heavy-handed I think | |
| 14:57:46 | mriedem | but the block device in my guest isn't resized to the new size, so i report a support ticket | |
| 14:58:17 | mriedem | the admin has to go figure out through the logs that the event was basically ignored on the ocata compute because it didn't recognize it | |
| 14:58:46 | knikolla_phone | jaypipes: i like dragons. | |
| 14:58:50 | dansmith | ah, yeah, but.. they triggered the resize from the cinder side right? | |
| 14:58:53 | mriedem | we also don't have any policy check for this to effectively disable it per event until your computes are all upgraded | |
| 14:58:58 | dansmith | mriedem: blocking the event from cinder isn't going to tell the user anything right? | |
| 14:58:58 | mriedem | dansmith: yes | |
| 14:59:06 | mriedem | but cinder is pike and the instance is runnign on an ocata compute | |
| 14:59:07 | knikolla_phone | We were left with an inconsistent db after upgrade. Some instances failed to create and delete. But everything else was fine. | |
| 14:59:33 | mriedem | dansmith: i was thinking that if we could detect in the api that this isn't going to work based on service version, then we return a 400 to cinder and cinder rolls back the volume resize | |
| 14:59:37 | mriedem | which is the indication to the user | |
| 14:59:45 | dansmith | mriedem: that's super expensive | |
| 15:00:00 | dansmith | mriedem: and.. can you even roll it back? often times block device resizes are one-way | |
| 15:00:06 | mriedem | alternatively, if cinder has a policy rule for extending an attached volume, then the admin can disable this from the cinder side until they know they can handle it | |
| 15:00:25 | mriedem | mgagne: does the cinder side of this have a unique policy rule for extending attached volumes? | |
| 15:00:34 | mriedem | or is it just the same as os-extend for 'available' non-attached volumes? | |
| 15:01:16 | jaypipes | knikolla_phone: here's a dragon for you: http://bit.ly/2sLVKtR | |
| 15:01:28 | mriedem | mgagne: because if i were a xen or vmware or hyperv shop, and this just will never work, i'd want to disable the cinder api behavior globally | |
| 15:02:28 | knikolla_phone | jaypipes: the nova-manage project quota_refresh_usage command didn't do anything. | |
| 15:02:56 | knikolla_phone | Newton -> ocata was the only problematic upgrade so far for us. :( | |
| 15:04:29 | jaypipes | knikolla_phone: other than manually going through your DB, I'm not entirely sure I can help you on this channel. There's some mismatch somewhere between the project_quota_usages or quota_usages table and the "true" resource tables like instances, instance_extra, pci_devices, etc | |
| 15:04:54 | jaypipes | knikolla_phone: is this isolated to a particular tenant? or general? | |
| 15:06:01 | knikolla_phone | jaypipes: no, that's okay, you've helped more than enough. This is only limited to one tenant. | |
| 15:06:27 | jaypipes | knikolla_phone: and only for their instances quota? | |