Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-17
16:55:17 mriedem shaner: would have to check if live migrations were tracked with migrations records in kilo
16:55:52 mriedem alaski: sort of https://review.openstack.org/#/c/420439/
16:55:58 mriedem there was much gnashing of teeth
16:56:12 mriedem alaski: https://review.openstack.org/#/c/420439/5/nova/cmd/manage.py@1122
16:56:37 mriedem basically we weren't sure about the original intent of using the api_database.connection for the cell0 netloc and name
16:56:53 mriedem but all agreed that nova_api_cell0 was a confusing db name for a db that is not using the api db schema
16:57:18 alaski yep
16:57:19 mriedem but also agreed that cell0 might not be on the same host as the api db, or main db,
16:57:38 mriedem and finally agreed that us trying to guess is a bad idea and people should just provide the value explicitly
16:57:45 alaski right, in a large deployment I expect all dbs to be separate
16:57:58 mriedem we probably need to make a note in our docs for ^
16:58:16 mriedem i.e. .. note:: we'll guess, but might be wrong, and then cell0 won't work, don't say we didn't tell you
16:59:02 mriedem sdague: so in devstack, if you're not on the subnode DATABASE_HOST isn't set,
16:59:13 mriedem sdague: and on the api (primary) node, it gets defaulted to localhost: DATABASE_HOST=MYSQL_HOST=SERVICE_LOCAL_HOST=127.0.0.1
16:59:23 alaski yeah. the simple_cell_setup command seems to sit in the same place as something like devstack, i.e. it'll get you setup and show you what needs doing but probably isn't what you want for production
16:59:42 mriedem sdague: but the devstack docs say that DATABASE_HOST=$SERVICE_HOST by default if not set, and that's also what's used for the MQ host URL
16:59:49 mriedem sdague: so anything you know of that would be bad for me trying to fix that?
17:00:33 alaski mriedem: so the original reason for colocating cell0 with the api db vs the main db is that there is only ever expected to be one api db and one cell0 db, but there could be multiple "main" dbs. So thinking of where cell0 is gets confusing when there's multiple
17:00:49 openstackgerrit Sivasathurappan Radhakrishnan proposed openstack/nova: Rollback on live migration failure https://review.openstack.org/379032
17:01:02 sdague mriedem: trying to establish context
17:01:09 mriedem well i guess https://github.com/openstack-dev/devstack/blob/master/doc/source/guides/multinode-lab.rst#configure-compute-nodes says MYSQL_HOST=$SERVICE_HOST
17:01:24 mriedem sdague: in devstack today we create cell mappings on the primary node
17:01:39 mriedem sdague: and i'm seeing that the db connection in those has 127.0.0.1 as the host
17:01:50 mriedem because MYSQL_HOST=${MYSQL_HOST:-$SERVICE_LOCAL_HOST}
17:02:32 mriedem sdague: and i've got a patch up to discover the subnode compute once n-cpu is running,
17:02:46 mriedem but that's failing because the subnode can't connect to the cell db on 127.0.0.1 b/c the db doesn't exist on the subnode
17:03:19 mriedem i'm actually confused as to how the subnode nova.conf has the proper database connection url
17:03:25 sdague mriedem: which services are on the subnode?
17:03:32 mriedem n-cpu
17:03:42 mriedem oh i know why the subnode nova.conf is correct
17:03:48 sdague I'm confused, n-cpu doesn't have a db connection url
17:03:50 mriedem because d-g passes in the DATABASE_HOST in localrc
17:03:58 mriedem it does in nova.conf
17:04:03 openstackgerrit Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell0 https://review.openstack.org/421384
17:04:15 sdague mriedem: ok, but n-cpu doesn't use it, right?
17:04:16 mriedem http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/subnode-2/etc/nova/nova.conf.txt.gz
17:04:20 mriedem nope
17:04:27 sdague because that's was the whole point of conductor :)
17:04:38 mriedem yeah
17:04:46 mriedem well, my change doesn't it need it for n-cpu
17:04:57 mriedem it needs it for the discover_hosts CLI to run and map the subnode compute host to the parent cell
17:05:14 mriedem but that parent cell is created when setting up the api node, which thinks it's db host ip is 127.0.0.1
17:05:27 mriedem anywho, i think it's a simple fix, i'll just push that up and we can see
17:05:57 mriedem it's a really weird mix of d-g doesn't do something for the primary node, but does for the subnode, and you have to keep that all in your head at the same time to know what's what
17:06:28 mriedem d-g should probably just always set DATABASE_HOST in localrc for multinode jobs
17:06:32 mriedem then devstack doesn't need to guess
17:06:49 owalsh mriedem, bauzas: hola, I've a question about the cell_v2 api db sync...
17:07:19 mriedem melwitt: you can continue doing all that you droppe doing from earlier pings, i've figured out the problem :)
17:07:23 mriedem owalsh: shoot
17:07:32 mriedem owalsh: first, that's not a thing
17:07:34 mriedem but ok
17:08:00 sdague mriedem: I think the DATABASE_HOST was jammed in for cinder
17:08:04 openstackgerrit int32bit proposed openstack/python-novaclient: Add CLI to show instance usage audit logs https://review.openstack.org/409413
17:08:08 owalsh mriedem: if a user installs the N tripleo undercloud but upgrades to O before adding any ironic nodes the sync will fail
17:08:20 owalsh mriedem: here https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L51
17:08:20 mriedem N = newton?
17:08:28 owalsh ^ yes
17:08:29 sdague and because it never broken anything else, it was never looked at
17:08:47 sdague mriedem: https://review.openstack.org/#/c/106043/ is where the subnode setting of that comes from
17:09:17 mriedem sdague: yeah, i could probably have a patch in both projects
17:09:17 mriedem sdague: yeah i know
17:09:17 mriedem i was in there yesterday
17:09:39 mriedem sdague: specifically this https://review.openstack.org/#/c/106043/23/devstack-vm-gate.sh@312
17:09:59 mriedem i'm saying we should just always set DATABASE_HOST in the localrc regardless of node
17:10:30 owalsh mriedem: question being is that check absolutely necessary?
17:11:01 owalsh mriedem: if a user has not run simple setup then it will fail a previous check. If map_cell0/create_cell etc... is being used then can we assume they know what they are doing
17:11:07 mriedem owalsh: that check assumed that if you were doing a fresh install there would be no flavors yet and it would opt out
17:11:13 mriedem *noop i mean
17:11:29 mriedem but if there were flavors, it's an upgrade and there would be computes
17:11:29 owalsh mriedem: ah, forgot to mention that installing the undercloud creates default flavors
17:11:48 mriedem gdi
17:11:51 mriedem can someone please reset the 'days since tripleo disaster' counter?
17:12:48 owalsh mriedem: :-)
17:12:52 mriedem so what are your steps again?
17:13:07 mriedem install newton, sync api db, sync nova db, create flavors,...
17:13:16 sdague mriedem: https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L468-L469 - probably that should be conditional behind n-api / n-cond / n-sched enabled
17:17:03 owalsh mriedem: install newton, create flavors (implicitly by installing TripleO undercloud), upgrade to ocata, create cell0 db, map cell0, create cell1, sync nova db, discover_hosts (none yet), sync api db ... deploy computes, discover_hosts
17:17:26 owalsh mriedem: I err think...
17:17:47 openstackgerrit Dan Peschman proposed openstack/nova: nova-manage cell_v2 map_cell0 exit 0 https://review.openstack.org/420132
17:18:28 mriedem owalsh: so between "discover_hosts (none yet), sync api db" you blow up
17:18:34 owalsh mriedem: yes
17:19:28 mriedem fudg
17:19:32 mriedem *e
17:20:38 mriedem see how nova-status handles this exact same thing https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L153-L171
17:20:42 mriedem it doesn't consider it a failure
17:22:15 owalsh mriedem: yea, I'm thinking the db migraion could be relaxed. It will fail on the first check if the user hasn't run simple_cell_setup
17:23:24 owalsh mriedem: if it gets as far as the host mapping check it implies that map_cell0 and create_cell were used, or simple_cell_setup failed...
17:28:27 mriedem i need to reset my brain on how we can even add cell mappings before running the api db sync
17:29:53 mriedem in devstack we create the nova_api db and then sync it here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L694
17:30:11 mriedem then later we run simple_cell_setup here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L950
17:30:28 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209
17:30:28 mriedem oh devstack works because when we sync the api_db we don't have any flavors yet
17:30:32 mriedem so it noops
17:31:38 mriedem ok let me wrap up this thing i'm fixing first, then i can come back to this
17:34:10 gibi mriedem: hi! I saw you patch chasing the race in the notification sample test https://review.openstack.org/#/c/421221
17:34:25 mriedem yeah
17:34:27 gibi mriedem: do we want to merge that to see more logs or just use to locally reproduce the problem?
17:34:35 mriedem gibi: more logs
17:34:42 mriedem and so i can fingerprint the failure in elastic-recheck
17:35:04 gibi mriedem: OK, then I'm totally +1 on that

Earlier   Later