| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 14:02:50 | dansmith | that's what I would expect if you're all pointing at the other cell | |
| 14:02:54 | dansmith | but I'm not sure how we're hitting that not unique error then | |
| 14:02:58 | dansmith | oh, | |
| 14:03:04 | edleafe | alex_xu: I would mimic what was done for custom resources, but using the 'traits:' prefix. Not too sure how to distinguish required vs. preferred | |
| 14:03:05 | dansmith | unless you have two cell records for the same cell? | |
| 14:03:26 | dansmith | dtantsur: also in nova_api, dump out the cell_mapping table please | |
| 14:03:39 | dtantsur | sure, gimme a second | |
| 14:03:56 | dtantsur | I need to figure out how to escape column names in mysql (binary is a reserved word, apparently) | |
| 14:04:16 | alex_xu | edleafe: but really want to understand the reason why we continue put something in the extra spec first | |
| 14:04:21 | dansmith | dtantsur: heh | |
| 14:04:51 | dansmith | dtantsur: I think if you use the full path it'll work, i.e.: SELECT services.binary FROM services | |
| 14:05:47 | dtantsur | dansmith: thnx! ok, so here are services: https://paste.fedoraproject.org/paste/ZK~YdTqfsWnpIImEF6TxLQ | |
| 14:06:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810 | |
| 14:06:43 | sfinucan | edleafe, jaypipes: ^ probably not suitable for merge yet, but there you go | |
| 14:06:58 | jaypipes | sfinucan: hey, cool, thanks for working on that | |
| 14:07:59 | dtantsur | dansmith: cell_mappings has 2 cells | |
| 14:08:23 | dansmith | dtantsur: do they both point to the same database_connection ? | |
| 14:08:59 | dtantsur | dansmith: nope | |
| 14:09:19 | dansmith | well, then I'm not sure how you're getting that exception | |
| 14:09:41 | openstackgerrit | Maciej Kucia proposed openstack/nova master: [doc] Updated sqlalchemy URL in migrate readme https://review.openstack.org/472549 | |
| 14:09:56 | dansmith | dtantsur: is this a machine I can get on to add some debug? or can you just sprinkle some debug statements in that _find_service_in_cell() function? | |
| 14:10:31 | dtantsur | dansmith: I can do both, the latter is easier, I guess | |
| 14:10:40 | dtantsur | just after the current meeting | |
| 14:12:51 | dansmith | hmm | |
| 14:13:06 | edleafe | alex_xu: because right now flavors are the only means to specify what you want Nova to build. Until we can clean up that technical debt, we are stuck using flavors, which means cramming more and more stuff into extra_specs | |
| 14:15:25 | dtantsur | dansmith: what exactly would you like to see printed? I think I have an idea, but just in case you have something else in mind | |
| 14:16:50 | dansmith | dtantsur: that for loop should iterate twice.. once over each cell. if cell0 is empty, it should never set "service" to something, and thus in the second/last iteration, it should not hit the ServiceNotUnique raise, because service hasn't been set yet | |
| 14:17:01 | dansmith | dtantsur: so just anything around service getting set, what it's set to, etc | |
| 14:17:13 | dtantsur | yep, this is my assumption too. I'll try to check it. | |
| 14:23:18 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:24:28 | sdague | melwitt / mriedem - in https://review.openstack.org/#/c/446247/23 ... auto_allocated floating ips are excluded from quota? | |
| 14:24:33 | sdague | is that they way it's always been? | |
| 14:26:20 | dtantsur | dansmith: it looks like both Cell objects (cell0 and default) have the same database connection (poining to "nova") Oo | |
| 14:27:04 | dansmith | dtantsur: okay, that makes sense for getting that not unique error, so now we need to figure out why the db is different | |
| 14:27:10 | dtantsur | wait, no | |
| 14:27:29 | sdague | ah, I found other code that indicates yes | |
| 14:27:41 | sdague | mriedem: ok, I went up until the first jenkins failing patch | |
| 14:27:54 | mriedem | sdague: that's the way it's always been | |
| 14:27:56 | sdague | and they look good | |
| 14:28:11 | sdague | mriedem: yeh, I didn't realize, but the follow on patch touched a line where it was more obvious | |
| 14:29:09 | dtantsur | dansmith: sorry for confusing. the cells are ok. I'm just getting the same host out of both.. | |
| 14:29:09 | mriedem | thanks for hitting those | |
| 14:29:59 | dansmith | dtantsur: okay | |
| 14:31:13 | dtantsur | but 'services' table is empty on nova_cell0, I've just double-checked >_< | |
| 14:31:44 | sdague | mriedem: my guess is that got caught up in the 6 way octopus merge limits with git | |
| 14:32:01 | sdague | mriedem: so honestly rebasing everything once the base set lands is probably in order anyway | |
| 14:32:36 | dansmith | dtantsur: and just to be clear, the api_connection is pointed at nova_api and not nova_placement right? | |
| 14:32:46 | dtantsur | lemme check too | |
| 14:33:09 | sdague | mriedem: my only wish in those patches (which is long past as it landed in march) is that the config option was strict_quotas instead of recheck_quotas, just from a nice name perspective | |
| 14:34:31 | dtantsur | dansmith: [api_database]connection points to nova_api | |
| 14:34:59 | dansmith | dtantsur: okay so in that for loop, is it finding/setting | |
| 14:35:06 | dansmith | service to anything on the cell0 pass? | |
| 14:35:30 | dtantsur | dansmith: yep. lemme show you the output of my added logging | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: explain payload inheritance in notification devref https://review.openstack.org/453667 | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add snapshot id to the snapshot notifications https://review.openstack.org/453077 | |
| 14:36:30 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:36:48 | knikolla_phone | How does nova calculate quota usage? After manually setting an instance as deleted in the database 'instances' table the resources still show up as used. | |
| 14:37:08 | dansmith | heh | |
| 14:37:12 | dtantsur | dansmith: https://paste.fedoraproject.org/paste/r26SqCAGrk8dXPlOk-sfjg | |
| 14:39:27 | dansmith | dtantsur: that's kinda hard to make sense of, especially without seeing the code, but.. why the HostMappingNotFound messages interleaved with the iteration of that for loop? | |
| 14:39:51 | dtantsur | dansmith: I think it was a bug in oslo.log | |
| 14:39:57 | dansmith | oh eff me | |
| 14:40:00 | dansmith | dtantsur: I see the bug | |
| 14:41:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using targeted cell context when finding services in cells https://review.openstack.org/475830 | |
| 14:41:30 | dansmith | dtantsur: try that ^ :) | |
| 14:41:37 | dtantsur | thanks! | |
| 14:42:19 | dansmith | dtantsur: it was iterating but not actually looking in the other database :P | |
| 14:42:27 | dtantsur | I see | |
| 14:46:33 | dansmith | dtantsur: if this fixes you, then: sorry for wasting your time | |
| 14:47:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/475600 | |
| 14:48:04 | dtantsur | dansmith: w00t, it works with this patch :) thanks! | |
| 14:48:24 | dansmith | dtantsur: cool, I shall write a test and then try to get that in asap | |
| 14:48:31 | dtantsur | great | |
| 14:48:39 | jaypipes | knikolla_phone: you need to run the nova-manage quota refresh command. | |
| 14:48:50 | dansmith | cc: mriedem | |
| 14:49:03 | jaypipes | knikolla_phone: unfortunately, quotas uses different tables than instances to store usage. | |
| 14:49:20 | dansmith | jaypipes: yesterday that was true | |
| 14:49:23 | dansmith | jaypipes: today it may not be | |
| 14:49:25 | jaypipes | wait, what? | |
| 14:49:28 | jaypipes | oh, yeah. | |
| 14:49:40 | jaypipes | but pretty sure knikolla_phone doesn't have yesterday's code deployed ;) | |
| 14:49:45 | dansmith | GASP | |
| 14:50:05 | mriedem | hells bells https://review.openstack.org/#/c/475830/1/nova/compute/api.py | |
| 14:50:11 | dansmith | jaypipes: I know, but it's amusing timing given the flurry of them in the gate atm | |
| 14:50:19 | knikolla_phone | Jaypipes: we're running ocata | |
| 14:50:36 | jaypipes | knikolla_phone: yup, so you need to do the nova-manage quota refresh thing :) | |
| 14:50:41 | jaypipes | dansmith: ack | |
| 14:51:29 | knikolla_phone | Jaypipes: thanks. Will give it a try now. | |
| 14:53:00 | mriedem | mgagne: sorry it took awhile, but i finally got another thorough review on https://review.openstack.org/#/c/454322/ | |
| 14:53:10 | mriedem | mgagne: lots of edge cases in there that need to be dealt with | |
| 14:53:23 | mriedem | dansmith: so, | |
| 14:53:34 | mgagne | ok, will read and address them asap | |
| 14:53:37 | mriedem | something i've been meaning to pester other cores about for their opinions on https://review.openstack.org/#/c/454322/ | |
| 14:53:55 | mriedem | dansmith: ^ is a new compute driver capability, | |
| 14:54:05 | mriedem | called by cinder when a volume is resized | |
| 14:54:21 | mriedem | if you're doing this on an instance running on an ocata compute, that's going to be ignored | |
| 14:54:30 | mriedem | with no indication to the user/admin | |
| 14:55:06 | knikolla_phone | jaypipes: that didn't work. Still ghost resources. | |
| 14:55:26 | mriedem | dansmith: well, take that back, you'll get a warning in the compute logs on the ocata node for "Received unexpected event %(event)s for instance" | |
| 14:55:31 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform keypair.delete notification https://review.openstack.org/463946 | |