| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-20 | |||
| 13:37:52 | mriedem | only thing left in os-hypervisors at the top is docs and paging | |
| 13:38:14 | dtantsur | does it mean that I'll have to stop referring to host names when adding them to aggregates? please pardon my ignorance, I'm not sure how it solves the problem in question | |
| 13:38:16 | dansmith | he needs services | |
| 13:38:39 | dansmith | dtantsur: oh are you getting that while querying for a service by hostname? | |
| 13:39:08 | dtantsur | dansmith: I'm doing $ openstack aggregate add host baremetal-hosts overcloud-controller-0.localdomain | |
| 13:39:20 | dtantsur | do add the ironic host to an aggregate | |
| 13:39:38 | dansmith | I see | |
| 13:39:40 | dtantsur | which may be not a perfect approach to a problem of hybrid bm-vm environment, but that's what we do now | |
| 13:39:42 | dansmith | let me look a bit | |
| 13:40:26 | dansmith | ohhh | |
| 13:40:42 | dtantsur | I don't like this "ohhh" :) | |
| 13:40:43 | dansmith | because you have multiple per host maybe | |
| 13:40:57 | dansmith | hang on, I have no coffee in my system. this is not good. | |
| 13:41:05 | dtantsur | multiple what? I definitely have only one nova-compute process per host.. | |
| 13:41:09 | boolman | Hi guys, I'm having trouble with a new deployment.. I'm using juju to deploy ocata, so all the configuration is automated, and yes I have the [placement] section. and I seem to have issues with my compute-nodes/placement API, http://ix.io/xHP http://ix.io/xHO | |
| 13:41:19 | dtantsur | dansmith: sure, lack of coffee is an urgent priority bug :) | |
| 13:42:25 | dansmith | dtantsur: hmm, no this should work.. are you running control services on the same node as a compute? | |
| 13:42:33 | dtantsur | dansmith: I am | |
| 13:42:40 | dansmith | dtantsur: with different config files I assume? | |
| 13:42:54 | dtantsur | dansmith: I don't think so actually | |
| 13:43:12 | dansmith | dtantsur: if not then they wouldn't be in different cells... | |
| 13:43:32 | dtantsur | are cells assigned via nova.conf? | |
| 13:43:50 | dansmith | no, but which cell you check into is determined by which database url you have configured | |
| 13:44:32 | dtantsur | hmm, ok. these hosts may indeed end up in a wrong cell.. however, it does not quite explain to me why we have duplicate hosts exception | |
| 13:44:50 | dansmith | well, | |
| 13:44:51 | dansmith | if they | |
| 13:44:53 | dansmith | are in different cells, | |
| 13:45:28 | dansmith | and they both report the same hostname, then there are multiple services with the same hostname but in different cells | |
| 13:46:09 | dtantsur | they = who? for now it seems that I have nova-api, scheduler and ironic's computes in one cell, then real compute in another | |
| 13:46:18 | dansmith | right | |
| 13:46:30 | dansmith | they = any service (n-api, n-cpu, etc) | |
| 13:46:49 | dansmith | since they have the same hostname as what n-cpu is reporting, | |
| 13:47:13 | dansmith | we could look that service up to find the cell, but end up getting the wrong one | |
| 13:47:22 | dansmith | which is why we now check for dupes and fail if the result is ambiguous | |
| 13:47:41 | dansmith | dtantsur: can we check to see if they are using the same config and thus reporting into the same cell for the moment? | |
| 13:49:02 | dtantsur | dansmith: I have only one nova.conf (provided by puppet, I guess), so I think that's the case | |
| 13:49:11 | dtantsur | I still don't quite understand why the result is ambiguos | |
| 13:49:20 | dansmith | actually, that query should be looking specifically for a nova-compute service, which shouldn't be duplicated if you only have one compute | |
| 13:49:26 | dansmith | dtantsur: that's what we're trying to find out | |
| 13:49:30 | dtantsur | right | |
| 13:49:43 | dansmith | dtantsur: can you dump the services table for your cell0 and other cell databases for me? | |
| 13:49:55 | dtantsur | lemme try | |
| 13:51:07 | dtantsur | dansmith: there are 3 databases, I guess I need one that is "nova", not "nova_api" or "nova_placement". right? | |
| 13:52:43 | alex_xu | edleafe: not sure what I should do for required traits | |
| 13:53:01 | dansmith | dtantsur: you should have nova and nova_cell0 | |
| 13:54:02 | dtantsur | ok, lemme see | |
| 14:01:00 | dtantsur | dansmith: ok, cell0 is empty. lemme fetch another one for you | |
| 14:02:33 | dansmith | cell0.services you mean right? | |
| 14:02:50 | dansmith | that's what I would expect if you're all pointing at the other cell | |
| 14:02:54 | dansmith | but I'm not sure how we're hitting that not unique error then | |
| 14:02:58 | dansmith | oh, | |
| 14:03:04 | edleafe | alex_xu: I would mimic what was done for custom resources, but using the 'traits:' prefix. Not too sure how to distinguish required vs. preferred | |
| 14:03:05 | dansmith | unless you have two cell records for the same cell? | |
| 14:03:26 | dansmith | dtantsur: also in nova_api, dump out the cell_mapping table please | |
| 14:03:39 | dtantsur | sure, gimme a second | |
| 14:03:56 | dtantsur | I need to figure out how to escape column names in mysql (binary is a reserved word, apparently) | |
| 14:04:16 | alex_xu | edleafe: but really want to understand the reason why we continue put something in the extra spec first | |
| 14:04:21 | dansmith | dtantsur: heh | |
| 14:04:51 | dansmith | dtantsur: I think if you use the full path it'll work, i.e.: SELECT services.binary FROM services | |
| 14:05:47 | dtantsur | dansmith: thnx! ok, so here are services: https://paste.fedoraproject.org/paste/ZK~YdTqfsWnpIImEF6TxLQ | |
| 14:06:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document the scheduler workflow https://review.openstack.org/475810 | |
| 14:06:43 | sfinucan | edleafe, jaypipes: ^ probably not suitable for merge yet, but there you go | |
| 14:06:58 | jaypipes | sfinucan: hey, cool, thanks for working on that | |
| 14:07:59 | dtantsur | dansmith: cell_mappings has 2 cells | |
| 14:08:23 | dansmith | dtantsur: do they both point to the same database_connection ? | |
| 14:08:59 | dtantsur | dansmith: nope | |
| 14:09:19 | dansmith | well, then I'm not sure how you're getting that exception | |
| 14:09:41 | openstackgerrit | Maciej Kucia proposed openstack/nova master: [doc] Updated sqlalchemy URL in migrate readme https://review.openstack.org/472549 | |
| 14:09:56 | dansmith | dtantsur: is this a machine I can get on to add some debug? or can you just sprinkle some debug statements in that _find_service_in_cell() function? | |
| 14:10:31 | dtantsur | dansmith: I can do both, the latter is easier, I guess | |
| 14:10:40 | dtantsur | just after the current meeting | |
| 14:12:51 | dansmith | hmm | |
| 14:13:06 | edleafe | alex_xu: because right now flavors are the only means to specify what you want Nova to build. Until we can clean up that technical debt, we are stuck using flavors, which means cramming more and more stuff into extra_specs | |
| 14:15:25 | dtantsur | dansmith: what exactly would you like to see printed? I think I have an idea, but just in case you have something else in mind | |
| 14:16:50 | dansmith | dtantsur: that for loop should iterate twice.. once over each cell. if cell0 is empty, it should never set "service" to something, and thus in the second/last iteration, it should not hit the ServiceNotUnique raise, because service hasn't been set yet | |
| 14:17:01 | dansmith | dtantsur: so just anything around service getting set, what it's set to, etc | |
| 14:17:13 | dtantsur | yep, this is my assumption too. I'll try to check it. | |
| 14:23:18 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Allow tuple as for nics value https://review.openstack.org/475816 | |
| 14:24:28 | sdague | melwitt / mriedem - in https://review.openstack.org/#/c/446247/23 ... auto_allocated floating ips are excluded from quota? | |
| 14:24:33 | sdague | is that they way it's always been? | |
| 14:26:20 | dtantsur | dansmith: it looks like both Cell objects (cell0 and default) have the same database connection (poining to "nova") Oo | |
| 14:27:04 | dansmith | dtantsur: okay, that makes sense for getting that not unique error, so now we need to figure out why the db is different | |
| 14:27:10 | dtantsur | wait, no | |
| 14:27:29 | sdague | ah, I found other code that indicates yes | |
| 14:27:41 | sdague | mriedem: ok, I went up until the first jenkins failing patch | |
| 14:27:54 | mriedem | sdague: that's the way it's always been | |
| 14:27:56 | sdague | and they look good | |
| 14:28:11 | sdague | mriedem: yeh, I didn't realize, but the follow on patch touched a line where it was more obvious | |
| 14:29:09 | mriedem | thanks for hitting those | |
| 14:29:09 | dtantsur | dansmith: sorry for confusing. the cells are ok. I'm just getting the same host out of both.. | |
| 14:29:59 | dansmith | dtantsur: okay | |
| 14:31:13 | dtantsur | but 'services' table is empty on nova_cell0, I've just double-checked >_< | |
| 14:31:44 | sdague | mriedem: my guess is that got caught up in the 6 way octopus merge limits with git | |
| 14:32:01 | sdague | mriedem: so honestly rebasing everything once the base set lands is probably in order anyway | |
| 14:32:36 | dansmith | dtantsur: and just to be clear, the api_connection is pointed at nova_api and not nova_placement right? | |
| 14:32:46 | dtantsur | lemme check too | |
| 14:33:09 | sdague | mriedem: my only wish in those patches (which is long past as it landed in march) is that the config option was strict_quotas instead of recheck_quotas, just from a nice name perspective | |
| 14:34:31 | dtantsur | dansmith: [api_database]connection points to nova_api | |
| 14:34:59 | dansmith | dtantsur: okay so in that for loop, is it finding/setting | |
| 14:35:06 | dansmith | service to anything on the cell0 pass? | |