| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 14:10:31 | dtantsur | dansmith: I can do both, the latter is easier, I guess | |
| 14:10:40 | dtantsur | just after the current meeting | |
| 14:12:51 | dansmith | hmm | |
| 14:13:06 | edleafe | alex_xu: because right now flavors are the only means to specify what you want Nova to build. Until we can clean up that technical debt, we are stuck using flavors, which means cramming more and more stuff into extra_specs | |
| 14:15:25 | dtantsur | dansmith: what exactly would you like to see printed? I think I have an idea, but just in case you have something else in mind | |
| 14:16:50 | dansmith | dtantsur: that for loop should iterate twice.. once over each cell. if cell0 is empty, it should never set "service" to something, and thus in the second/last iteration, it should not hit the ServiceNotUnique raise, because service hasn't been set yet | |
| 14:17:01 | dansmith | dtantsur: so just anything around service getting set, what it's set to, etc | |
| 14:17:13 | dtantsur | yep, this is my assumption too. I'll try to check it. | |
| 14:23:18 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:24:28 | sdague | melwitt / mriedem - in https://review.openstack.org/#/c/446247/23 ... auto_allocated floating ips are excluded from quota? | |
| 14:24:33 | sdague | is that they way it's always been? | |
| 14:26:20 | dtantsur | dansmith: it looks like both Cell objects (cell0 and default) have the same database connection (poining to "nova") Oo | |
| 14:27:04 | dansmith | dtantsur: okay, that makes sense for getting that not unique error, so now we need to figure out why the db is different | |
| 14:27:10 | dtantsur | wait, no | |
| 14:27:29 | sdague | ah, I found other code that indicates yes | |
| 14:27:41 | sdague | mriedem: ok, I went up until the first jenkins failing patch | |
| 14:27:54 | mriedem | sdague: that's the way it's always been | |
| 14:27:56 | sdague | and they look good | |
| 14:28:11 | sdague | mriedem: yeh, I didn't realize, but the follow on patch touched a line where it was more obvious | |
| 14:29:09 | dtantsur | dansmith: sorry for confusing. the cells are ok. I'm just getting the same host out of both.. | |
| 14:29:09 | mriedem | thanks for hitting those | |
| 14:29:59 | dansmith | dtantsur: okay | |
| 14:31:13 | dtantsur | but 'services' table is empty on nova_cell0, I've just double-checked >_< | |
| 14:31:44 | sdague | mriedem: my guess is that got caught up in the 6 way octopus merge limits with git | |
| 14:32:01 | sdague | mriedem: so honestly rebasing everything once the base set lands is probably in order anyway | |
| 14:32:36 | dansmith | dtantsur: and just to be clear, the api_connection is pointed at nova_api and not nova_placement right? | |
| 14:32:46 | dtantsur | lemme check too | |
| 14:33:09 | sdague | mriedem: my only wish in those patches (which is long past as it landed in march) is that the config option was strict_quotas instead of recheck_quotas, just from a nice name perspective | |
| 14:34:31 | dtantsur | dansmith: [api_database]connection points to nova_api | |
| 14:34:59 | dansmith | dtantsur: okay so in that for loop, is it finding/setting | |
| 14:35:06 | dansmith | service to anything on the cell0 pass? | |
| 14:35:30 | dtantsur | dansmith: yep. lemme show you the output of my added logging | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: explain payload inheritance in notification devref https://review.openstack.org/453667 | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add snapshot id to the snapshot notifications https://review.openstack.org/453077 | |
| 14:36:30 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:36:48 | knikolla_phone | How does nova calculate quota usage? After manually setting an instance as deleted in the database 'instances' table the resources still show up as used. | |
| 14:37:08 | dansmith | heh | |
| 14:37:12 | dtantsur | dansmith: https://paste.fedoraproject.org/paste/r26SqCAGrk8dXPlOk-sfjg | |
| 14:39:27 | dansmith | dtantsur: that's kinda hard to make sense of, especially without seeing the code, but.. why the HostMappingNotFound messages interleaved with the iteration of that for loop? | |
| 14:39:51 | dtantsur | dansmith: I think it was a bug in oslo.log | |
| 14:39:57 | dansmith | oh eff me | |
| 14:40:00 | dansmith | dtantsur: I see the bug | |
| 14:41:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using targeted cell context when finding services in cells https://review.openstack.org/475830 | |
| 14:41:30 | dansmith | dtantsur: try that ^ :) | |
| 14:41:37 | dtantsur | thanks! | |
| 14:42:19 | dansmith | dtantsur: it was iterating but not actually looking in the other database :P | |
| 14:42:27 | dtantsur | I see | |
| 14:46:33 | dansmith | dtantsur: if this fixes you, then: sorry for wasting your time | |
| 14:47:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/475600 | |
| 14:48:04 | dtantsur | dansmith: w00t, it works with this patch :) thanks! | |
| 14:48:24 | dansmith | dtantsur: cool, I shall write a test and then try to get that in asap | |
| 14:48:31 | dtantsur | great | |
| 14:48:39 | jaypipes | knikolla_phone: you need to run the nova-manage quota refresh command. | |
| 14:48:50 | dansmith | cc: mriedem | |
| 14:49:03 | jaypipes | knikolla_phone: unfortunately, quotas uses different tables than instances to store usage. | |
| 14:49:20 | dansmith | jaypipes: yesterday that was true | |
| 14:49:23 | dansmith | jaypipes: today it may not be | |
| 14:49:25 | jaypipes | wait, what? | |
| 14:49:28 | jaypipes | oh, yeah. | |
| 14:49:40 | jaypipes | but pretty sure knikolla_phone doesn't have yesterday's code deployed ;) | |
| 14:49:45 | dansmith | GASP | |
| 14:50:05 | mriedem | hells bells https://review.openstack.org/#/c/475830/1/nova/compute/api.py | |
| 14:50:11 | dansmith | jaypipes: I know, but it's amusing timing given the flurry of them in the gate atm | |
| 14:50:19 | knikolla_phone | Jaypipes: we're running ocata | |
| 14:50:36 | jaypipes | knikolla_phone: yup, so you need to do the nova-manage quota refresh thing :) | |
| 14:50:41 | jaypipes | dansmith: ack | |
| 14:51:29 | knikolla_phone | Jaypipes: thanks. Will give it a try now. | |
| 14:53:00 | mriedem | mgagne: sorry it took awhile, but i finally got another thorough review on https://review.openstack.org/#/c/454322/ | |
| 14:53:10 | mriedem | mgagne: lots of edge cases in there that need to be dealt with | |
| 14:53:23 | mriedem | dansmith: so, | |
| 14:53:34 | mgagne | ok, will read and address them asap | |
| 14:53:37 | mriedem | something i've been meaning to pester other cores about for their opinions on https://review.openstack.org/#/c/454322/ | |
| 14:53:55 | mriedem | dansmith: ^ is a new compute driver capability, | |
| 14:54:05 | mriedem | called by cinder when a volume is resized | |
| 14:54:21 | mriedem | if you're doing this on an instance running on an ocata compute, that's going to be ignored | |
| 14:54:30 | mriedem | with no indication to the user/admin | |
| 14:55:06 | knikolla_phone | jaypipes: that didn't work. Still ghost resources. | |
| 14:55:26 | mriedem | dansmith: well, take that back, you'll get a warning in the compute logs on the ocata node for "Received unexpected event %(event)s for instance" | |
| 14:55:31 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform keypair.delete notification https://review.openstack.org/463946 | |
| 14:56:07 | knikolla_phone | I know i asked this once and mriedem answered. But what does the deleted column in the table need to be set for a deleted instance? I can't remember what it was, sorry to ask again :( | |
| 14:56:09 | mriedem | dansmith: i was pondering doing a minimum compute service version check in the api for these types of events, since we know that you need a minimum of pike to support those events, and we could return a 400 to cinder otherwise, | |
| 14:56:16 | mriedem | and they could rollback the volume resize on their side | |
| 14:56:24 | jaypipes | knikolla_phone: to its ID value. | |
| 14:56:34 | mriedem | knikolla_phone: instances.deleted = instances.id | |
| 14:56:36 | mriedem | if it's deleted | |
| 14:56:53 | jaypipes | knikolla_phone: but honestly, you shouldn't be setting stuff manually in the DB. Dragons be there. | |
| 14:56:55 | mriedem | dansmith: but i'm wondering if that's too heavy handed, doing the service version check in the api | |
| 14:57:03 | dansmith | mriedem: yeah, I mean, it's a warning and the assumption was that we'd add events over time, so the warning isn't terrible | |
| 14:57:17 | dansmith | mriedem: just depends on whether or not we have different behavior before/after | |
| 14:57:32 | dansmith | if it's just an optimization that will continue to work but slower, then no need to be heavy-handed I think | |
| 14:57:32 | mriedem | my thinking was as a user i do this volume resize thing, and it all says it's fine, | |
| 14:57:46 | mriedem | but the block device in my guest isn't resized to the new size, so i report a support ticket | |
| 14:58:17 | mriedem | the admin has to go figure out through the logs that the event was basically ignored on the ocata compute because it didn't recognize it | |
| 14:58:46 | knikolla_phone | jaypipes: i like dragons. | |
| 14:58:50 | dansmith | ah, yeah, but.. they triggered the resize from the cinder side right? | |
| 14:58:53 | mriedem | we also don't have any policy check for this to effectively disable it per event until your computes are all upgraded | |
| 14:58:58 | mriedem | dansmith: yes | |
| 14:58:58 | dansmith | mriedem: blocking the event from cinder isn't going to tell the user anything right? | |
| 14:59:06 | mriedem | but cinder is pike and the instance is runnign on an ocata compute | |
| 14:59:07 | knikolla_phone | We were left with an inconsistent db after upgrade. Some instances failed to create and delete. But everything else was fine. | |