| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 13:44:53 | dansmith | are in different cells, | |
| 13:45:28 | dansmith | and they both report the same hostname, then there are multiple services with the same hostname but in different cells | |
| 13:46:09 | dtantsur | they = who? for now it seems that I have nova-api, scheduler and ironic's computes in one cell, then real compute in another | |
| 13:46:18 | dansmith | right | |
| 13:46:30 | dansmith | they = any service (n-api, n-cpu, etc) | |
| 13:46:49 | dansmith | since they have the same hostname as what n-cpu is reporting, | |
| 13:47:13 | dansmith | we could look that service up to find the cell, but end up getting the wrong one | |
| 13:47:22 | dansmith | which is why we now check for dupes and fail if the result is ambiguous | |
| 13:47:41 | dansmith | dtantsur: can we check to see if they are using the same config and thus reporting into the same cell for the moment? | |
| 13:49:02 | dtantsur | dansmith: I have only one nova.conf (provided by puppet, I guess), so I think that's the case | |
| 13:49:11 | dtantsur | I still don't quite understand why the result is ambiguos | |
| 13:49:20 | dansmith | actually, that query should be looking specifically for a nova-compute service, which shouldn't be duplicated if you only have one compute | |
| 13:49:26 | dansmith | dtantsur: that's what we're trying to find out | |
| 13:49:30 | dtantsur | right | |
| 13:49:43 | dansmith | dtantsur: can you dump the services table for your cell0 and other cell databases for me? | |
| 13:49:55 | dtantsur | lemme try | |
| 13:51:07 | dtantsur | dansmith: there are 3 databases, I guess I need one that is "nova", not "nova_api" or "nova_placement". right? | |
| 13:52:43 | alex_xu | edleafe: not sure what I should do for required traits | |
| 13:53:01 | dansmith | dtantsur: you should have nova and nova_cell0 | |
| 13:54:02 | dtantsur | ok, lemme see | |
| 14:01:00 | dtantsur | dansmith: ok, cell0 is empty. lemme fetch another one for you | |
| 14:02:33 | dansmith | cell0.services you mean right? | |
| 14:02:50 | dansmith | that's what I would expect if you're all pointing at the other cell | |
| 14:02:54 | dansmith | but I'm not sure how we're hitting that not unique error then | |
| 14:02:58 | dansmith | oh, | |
| 14:03:04 | edleafe | alex_xu: I would mimic what was done for custom resources, but using the 'traits:' prefix. Not too sure how to distinguish required vs. preferred | |
| 14:03:05 | dansmith | unless you have two cell records for the same cell? | |
| 14:03:26 | dansmith | dtantsur: also in nova_api, dump out the cell_mapping table please | |
| 14:03:39 | dtantsur | sure, gimme a second | |
| 14:03:56 | dtantsur | I need to figure out how to escape column names in mysql (binary is a reserved word, apparently) | |
| 14:04:16 | alex_xu | edleafe: but really want to understand the reason why we continue put something in the extra spec first | |
| 14:04:21 | dansmith | dtantsur: heh | |
| 14:04:51 | dansmith | dtantsur: I think if you use the full path it'll work, i.e.: SELECT services.binary FROM services | |
| 14:05:47 | dtantsur | dansmith: thnx! ok, so here are services: https://paste.fedoraproject.org/paste/ZK~YdTqfsWnpIImEF6TxLQ | |
| 14:06:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810 | |
| 14:06:43 | sfinucan | edleafe, jaypipes: ^ probably not suitable for merge yet, but there you go | |
| 14:06:58 | jaypipes | sfinucan: hey, cool, thanks for working on that | |
| 14:07:59 | dtantsur | dansmith: cell_mappings has 2 cells | |
| 14:08:23 | dansmith | dtantsur: do they both point to the same database_connection ? | |
| 14:08:59 | dtantsur | dansmith: nope | |
| 14:09:19 | dansmith | well, then I'm not sure how you're getting that exception | |
| 14:09:41 | openstackgerrit | Maciej Kucia proposed openstack/nova master: [doc] Updated sqlalchemy URL in migrate readme https://review.openstack.org/472549 | |
| 14:09:56 | dansmith | dtantsur: is this a machine I can get on to add some debug? or can you just sprinkle some debug statements in that _find_service_in_cell() function? | |
| 14:10:31 | dtantsur | dansmith: I can do both, the latter is easier, I guess | |
| 14:10:40 | dtantsur | just after the current meeting | |
| 14:12:51 | dansmith | hmm | |
| 14:13:06 | edleafe | alex_xu: because right now flavors are the only means to specify what you want Nova to build. Until we can clean up that technical debt, we are stuck using flavors, which means cramming more and more stuff into extra_specs | |
| 14:15:25 | dtantsur | dansmith: what exactly would you like to see printed? I think I have an idea, but just in case you have something else in mind | |
| 14:16:50 | dansmith | dtantsur: that for loop should iterate twice.. once over each cell. if cell0 is empty, it should never set "service" to something, and thus in the second/last iteration, it should not hit the ServiceNotUnique raise, because service hasn't been set yet | |
| 14:17:01 | dansmith | dtantsur: so just anything around service getting set, what it's set to, etc | |
| 14:17:13 | dtantsur | yep, this is my assumption too. I'll try to check it. | |
| 14:23:18 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:24:28 | sdague | melwitt / mriedem - in https://review.openstack.org/#/c/446247/23 ... auto_allocated floating ips are excluded from quota? | |
| 14:24:33 | sdague | is that they way it's always been? | |
| 14:26:20 | dtantsur | dansmith: it looks like both Cell objects (cell0 and default) have the same database connection (poining to "nova") Oo | |
| 14:27:04 | dansmith | dtantsur: okay, that makes sense for getting that not unique error, so now we need to figure out why the db is different | |
| 14:27:10 | dtantsur | wait, no | |
| 14:27:29 | sdague | ah, I found other code that indicates yes | |
| 14:27:41 | sdague | mriedem: ok, I went up until the first jenkins failing patch | |
| 14:27:54 | mriedem | sdague: that's the way it's always been | |
| 14:27:56 | sdague | and they look good | |
| 14:28:11 | sdague | mriedem: yeh, I didn't realize, but the follow on patch touched a line where it was more obvious | |
| 14:29:09 | mriedem | thanks for hitting those | |
| 14:29:09 | dtantsur | dansmith: sorry for confusing. the cells are ok. I'm just getting the same host out of both.. | |
| 14:29:59 | dansmith | dtantsur: okay | |
| 14:31:13 | dtantsur | but 'services' table is empty on nova_cell0, I've just double-checked >_< | |
| 14:31:44 | sdague | mriedem: my guess is that got caught up in the 6 way octopus merge limits with git | |
| 14:32:01 | sdague | mriedem: so honestly rebasing everything once the base set lands is probably in order anyway | |
| 14:32:36 | dansmith | dtantsur: and just to be clear, the api_connection is pointed at nova_api and not nova_placement right? | |
| 14:32:46 | dtantsur | lemme check too | |
| 14:33:09 | sdague | mriedem: my only wish in those patches (which is long past as it landed in march) is that the config option was strict_quotas instead of recheck_quotas, just from a nice name perspective | |
| 14:34:31 | dtantsur | dansmith: [api_database]connection points to nova_api | |
| 14:34:59 | dansmith | dtantsur: okay so in that for loop, is it finding/setting | |
| 14:35:06 | dansmith | service to anything on the cell0 pass? | |
| 14:35:30 | dtantsur | dansmith: yep. lemme show you the output of my added logging | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add snapshot id to the snapshot notifications https://review.openstack.org/453077 | |
| 14:35:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: explain payload inheritance in notification devref https://review.openstack.org/453667 | |
| 14:36:30 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:36:48 | knikolla_phone | How does nova calculate quota usage? After manually setting an instance as deleted in the database 'instances' table the resources still show up as used. | |
| 14:37:08 | dansmith | heh | |
| 14:37:12 | dtantsur | dansmith: https://paste.fedoraproject.org/paste/r26SqCAGrk8dXPlOk-sfjg | |
| 14:39:27 | dansmith | dtantsur: that's kinda hard to make sense of, especially without seeing the code, but.. why the HostMappingNotFound messages interleaved with the iteration of that for loop? | |
| 14:39:51 | dtantsur | dansmith: I think it was a bug in oslo.log | |
| 14:39:57 | dansmith | oh eff me | |
| 14:40:00 | dansmith | dtantsur: I see the bug | |
| 14:41:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using targeted cell context when finding services in cells https://review.openstack.org/475830 | |
| 14:41:30 | dansmith | dtantsur: try that ^ :) | |
| 14:41:37 | dtantsur | thanks! | |
| 14:42:19 | dansmith | dtantsur: it was iterating but not actually looking in the other database :P | |
| 14:42:27 | dtantsur | I see | |
| 14:46:33 | dansmith | dtantsur: if this fixes you, then: sorry for wasting your time | |
| 14:47:13 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/475600 | |
| 14:48:04 | dtantsur | dansmith: w00t, it works with this patch :) thanks! | |
| 14:48:24 | dansmith | dtantsur: cool, I shall write a test and then try to get that in asap | |
| 14:48:31 | dtantsur | great | |
| 14:48:39 | jaypipes | knikolla_phone: you need to run the nova-manage quota refresh command. | |
| 14:48:50 | dansmith | cc: mriedem | |
| 14:49:03 | jaypipes | knikolla_phone: unfortunately, quotas uses different tables than instances to store usage. | |
| 14:49:20 | dansmith | jaypipes: yesterday that was true | |
| 14:49:23 | dansmith | jaypipes: today it may not be | |