| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-17 | |||
| 16:59:23 | alaski | yeah. the simple_cell_setup command seems to sit in the same place as something like devstack, i.e. it'll get you setup and show you what needs doing but probably isn't what you want for production | |
| 16:59:42 | mriedem | sdague: but the devstack docs say that DATABASE_HOST=$SERVICE_HOST by default if not set, and that's also what's used for the MQ host URL | |
| 16:59:49 | mriedem | sdague: so anything you know of that would be bad for me trying to fix that? | |
| 17:00:33 | alaski | mriedem: so the original reason for colocating cell0 with the api db vs the main db is that there is only ever expected to be one api db and one cell0 db, but there could be multiple "main" dbs. So thinking of where cell0 is gets confusing when there's multiple | |
| 17:00:49 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova: Rollback on live migration failure https://review.openstack.org/379032 | |
| 17:01:02 | sdague | mriedem: trying to establish context | |
| 17:01:09 | mriedem | well i guess https://github.com/openstack-dev/devstack/blob/master/doc/source/guides/multinode-lab.rst#configure-compute-nodes says MYSQL_HOST=$SERVICE_HOST | |
| 17:01:24 | mriedem | sdague: in devstack today we create cell mappings on the primary node | |
| 17:01:39 | mriedem | sdague: and i'm seeing that the db connection in those has 127.0.0.1 as the host | |
| 17:01:50 | mriedem | because MYSQL_HOST=${MYSQL_HOST:-$SERVICE_LOCAL_HOST} | |
| 17:02:32 | mriedem | sdague: and i've got a patch up to discover the subnode compute once n-cpu is running, | |
| 17:02:46 | mriedem | but that's failing because the subnode can't connect to the cell db on 127.0.0.1 b/c the db doesn't exist on the subnode | |
| 17:03:19 | mriedem | i'm actually confused as to how the subnode nova.conf has the proper database connection url | |
| 17:03:25 | sdague | mriedem: which services are on the subnode? | |
| 17:03:32 | mriedem | n-cpu | |
| 17:03:42 | mriedem | oh i know why the subnode nova.conf is correct | |
| 17:03:48 | sdague | I'm confused, n-cpu doesn't have a db connection url | |
| 17:03:50 | mriedem | because d-g passes in the DATABASE_HOST in localrc | |
| 17:03:58 | mriedem | it does in nova.conf | |
| 17:04:03 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell0 https://review.openstack.org/421384 | |
| 17:04:15 | sdague | mriedem: ok, but n-cpu doesn't use it, right? | |
| 17:04:16 | mriedem | http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/subnode-2/etc/nova/nova.conf.txt.gz | |
| 17:04:20 | mriedem | nope | |
| 17:04:27 | sdague | because that's was the whole point of conductor :) | |
| 17:04:38 | mriedem | yeah | |
| 17:04:46 | mriedem | well, my change doesn't it need it for n-cpu | |
| 17:04:57 | mriedem | it needs it for the discover_hosts CLI to run and map the subnode compute host to the parent cell | |
| 17:05:14 | mriedem | but that parent cell is created when setting up the api node, which thinks it's db host ip is 127.0.0.1 | |
| 17:05:27 | mriedem | anywho, i think it's a simple fix, i'll just push that up and we can see | |
| 17:05:57 | mriedem | it's a really weird mix of d-g doesn't do something for the primary node, but does for the subnode, and you have to keep that all in your head at the same time to know what's what | |
| 17:06:28 | mriedem | d-g should probably just always set DATABASE_HOST in localrc for multinode jobs | |
| 17:06:32 | mriedem | then devstack doesn't need to guess | |
| 17:06:49 | owalsh | mriedem, bauzas: hola, I've a question about the cell_v2 api db sync... | |
| 17:07:19 | mriedem | melwitt: you can continue doing all that you droppe doing from earlier pings, i've figured out the problem :) | |
| 17:07:23 | mriedem | owalsh: shoot | |
| 17:07:32 | mriedem | owalsh: first, that's not a thing | |
| 17:07:34 | mriedem | but ok | |
| 17:08:00 | sdague | mriedem: I think the DATABASE_HOST was jammed in for cinder | |
| 17:08:04 | openstackgerrit | int32bit proposed openstack/python-novaclient: Add CLI to show instance usage audit logs https://review.openstack.org/409413 | |
| 17:08:08 | owalsh | mriedem: if a user installs the N tripleo undercloud but upgrades to O before adding any ironic nodes the sync will fail | |
| 17:08:20 | owalsh | mriedem: here https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L51 | |
| 17:08:20 | mriedem | N = newton? | |
| 17:08:28 | owalsh | ^ yes | |
| 17:08:29 | sdague | and because it never broken anything else, it was never looked at | |
| 17:08:47 | sdague | mriedem: https://review.openstack.org/#/c/106043/ is where the subnode setting of that comes from | |
| 17:09:17 | mriedem | sdague: yeah, i could probably have a patch in both projects | |
| 17:09:17 | mriedem | sdague: yeah i know | |
| 17:09:17 | mriedem | i was in there yesterday | |
| 17:09:39 | mriedem | sdague: specifically this https://review.openstack.org/#/c/106043/23/devstack-vm-gate.sh@312 | |
| 17:09:59 | mriedem | i'm saying we should just always set DATABASE_HOST in the localrc regardless of node | |
| 17:10:30 | owalsh | mriedem: question being is that check absolutely necessary? | |
| 17:11:01 | owalsh | mriedem: if a user has not run simple setup then it will fail a previous check. If map_cell0/create_cell etc... is being used then can we assume they know what they are doing | |
| 17:11:07 | mriedem | owalsh: that check assumed that if you were doing a fresh install there would be no flavors yet and it would opt out | |
| 17:11:13 | mriedem | *noop i mean | |
| 17:11:29 | mriedem | but if there were flavors, it's an upgrade and there would be computes | |
| 17:11:29 | owalsh | mriedem: ah, forgot to mention that installing the undercloud creates default flavors | |
| 17:11:48 | mriedem | gdi | |
| 17:11:51 | mriedem | can someone please reset the 'days since tripleo disaster' counter? | |
| 17:12:48 | owalsh | mriedem: :-) | |
| 17:12:52 | mriedem | so what are your steps again? | |
| 17:13:07 | mriedem | install newton, sync api db, sync nova db, create flavors,... | |
| 17:13:16 | sdague | mriedem: https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L468-L469 - probably that should be conditional behind n-api / n-cond / n-sched enabled | |
| 17:17:03 | owalsh | mriedem: install newton, create flavors (implicitly by installing TripleO undercloud), upgrade to ocata, create cell0 db, map cell0, create cell1, sync nova db, discover_hosts (none yet), sync api db ... deploy computes, discover_hosts | |
| 17:17:26 | owalsh | mriedem: I err think... | |
| 17:17:47 | openstackgerrit | Dan Peschman proposed openstack/nova: nova-manage cell_v2 map_cell0 exit 0 https://review.openstack.org/420132 | |
| 17:18:28 | mriedem | owalsh: so between "discover_hosts (none yet), sync api db" you blow up | |
| 17:18:34 | owalsh | mriedem: yes | |
| 17:19:28 | mriedem | fudg | |
| 17:19:32 | mriedem | *e | |
| 17:20:38 | mriedem | see how nova-status handles this exact same thing https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L153-L171 | |
| 17:20:42 | mriedem | it doesn't consider it a failure | |
| 17:22:15 | owalsh | mriedem: yea, I'm thinking the db migraion could be relaxed. It will fail on the first check if the user hasn't run simple_cell_setup | |
| 17:23:24 | owalsh | mriedem: if it gets as far as the host mapping check it implies that map_cell0 and create_cell were used, or simple_cell_setup failed... | |
| 17:28:27 | mriedem | i need to reset my brain on how we can even add cell mappings before running the api db sync | |
| 17:29:53 | mriedem | in devstack we create the nova_api db and then sync it here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L694 | |
| 17:30:11 | mriedem | then later we run simple_cell_setup here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L950 | |
| 17:30:28 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 17:30:28 | mriedem | oh devstack works because when we sync the api_db we don't have any flavors yet | |
| 17:30:32 | mriedem | so it noops | |
| 17:31:38 | mriedem | ok let me wrap up this thing i'm fixing first, then i can come back to this | |
| 17:34:10 | gibi | mriedem: hi! I saw you patch chasing the race in the notification sample test https://review.openstack.org/#/c/421221 | |
| 17:34:25 | mriedem | yeah | |
| 17:34:27 | gibi | mriedem: do we want to merge that to see more logs or just use to locally reproduce the problem? | |
| 17:34:35 | mriedem | gibi: more logs | |
| 17:34:42 | mriedem | and so i can fingerprint the failure in elastic-recheck | |
| 17:35:04 | gibi | mriedem: OK, then I'm totally +1 on that | |
| 17:40:10 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_instances https://review.openstack.org/421404 | |
| 17:50:56 | melwitt | mriedem: sweet! glad I could help by not being around | |
| 17:51:25 | mriedem | melwitt: i think this is the culmination of 2 days of frustration https://review.openstack.org/#/c/421399/1 | |
| 17:51:37 | mriedem | and about 4 hours of sleep | |
| 17:52:00 | dims | mriedem : wow one liner! | |
| 17:52:07 | melwitt | oof | |
| 17:53:42 | mriedem | so in devstack runs we create flavors basically last | |
| 17:53:54 | mriedem | we create the dbs and sync them, | |
| 17:53:58 | mriedem | we run simple_cell_setup | |
| 17:54:01 | mriedem | then we create the flavors | |
| 17:54:34 | mriedem | and that's how devstack avoids failing the api db migration for there being no host mappings (well we'd have host mappings after running simple_cell_setup) | |
| 17:55:18 | owalsh | ... and the api db doesn't know about hosts etc... so the flavor is the trigger yea | |
| 17:55:22 | mriedem | in any other fresh install, you can't create flavors in the api db until you sync the api db | |
| 17:55:57 | owalsh | sequence above is an upgrade, so we have the tables from newton I expect | |