Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-17
16:28:12 shaner mriedem, shouldn't 'migration-list' list all types of migrations though?
16:28:59 mriedem i don't think until 2.23 http://docs.openstack.org/developer/nova/api_microversion_history.html#id21
16:29:23 mriedem fwiw this is the api code in kilo https://github.com/openstack/nova/blob/kilo-eol/nova/api/openstack/compute/contrib/migrations.py#L53
16:30:00 openstackgerrit int32bit proposed openstack/nova: Choose VMware datastore by the least provisioned space https://review.openstack.org/402632
16:36:36 mriedem shaner: was the migration in progress or complete? if the migration was complete i wonder if the migration record was deleted
16:36:50 mriedem at this point kilo is pretty far eol so it would take some investigating
16:37:47 shaner mriedem, it seems either way, in-progress/complete, it doesn't show up
16:37:58 shaner I've tested it several times to no avail
16:38:13 shaner appreciate the info!
16:38:46 mriedem have you looked at the migrations table in the nova db?
16:39:07 mriedem are there any records where deleted = 0?
16:39:39 shaner mriedem, let me check
16:43:13 mriedem melwitt: hmm this is interesting
16:43:28 mriedem the db connection for the cell0 mapping in this failed run has localhost in it http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/devstacklog.txt.gz#_2017-01-17_14_43_53_943
16:43:44 mriedem nova-manage cell_v2 map_cell0 --database_connection 'mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8'
16:45:15 mriedem when we call simple_cell_setup the transport_url has a valid ip in it nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@10.20.124.3:5672/
16:45:40 andreykurilin mriedem: hi! I remember you spent a bit of time investigation an issue with novaclient's gate failures (which was fixed by blocking openstacksdk 0.9.12). If you have any details please share them here - https://review.openstack.org/#/c/418663
16:47:28 mriedem andreykurilin: done
16:47:34 andreykurilin mriedem: thanks
16:50:12 mriedem hmm wtf http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/devstacklog.txt.gz#_2017-01-17_14_27_19_600
16:50:19 mriedem BASE_SQL_CONN=mysql+pymysql://root:secretmysql@127.0.0.1
16:53:18 alaski mriedem: yesterday was a holiday for me so I missed your message. Thanks for not letting a national figure distract from the important birthday
16:54:07 mriedem alaski: you're MY national figure
16:54:13 shaner mriedem, all items in migrations db are cold migrations and indeed have 'deleted = 0'
16:54:44 shaner I don't see any from the past live-migrations I've done, so it seems, a live-migration isn't updating this table?
16:54:45 alaski mriedem: :)
16:54:50 mriedem DATABASE_HOST=MYSQL_HOST=SERVICE_LOCAL_HOST=127.0.0.1 blarg!
16:54:56 alaski mriedem: I see there was a question of naming, did that get resolved?
16:55:17 mriedem shaner: would have to check if live migrations were tracked with migrations records in kilo
16:55:52 mriedem alaski: sort of https://review.openstack.org/#/c/420439/
16:55:58 mriedem there was much gnashing of teeth
16:56:12 mriedem alaski: https://review.openstack.org/#/c/420439/5/nova/cmd/manage.py@1122
16:56:37 mriedem basically we weren't sure about the original intent of using the api_database.connection for the cell0 netloc and name
16:56:53 mriedem but all agreed that nova_api_cell0 was a confusing db name for a db that is not using the api db schema
16:57:18 alaski yep
16:57:19 mriedem but also agreed that cell0 might not be on the same host as the api db, or main db,
16:57:38 mriedem and finally agreed that us trying to guess is a bad idea and people should just provide the value explicitly
16:57:45 alaski right, in a large deployment I expect all dbs to be separate
16:57:58 mriedem we probably need to make a note in our docs for ^
16:58:16 mriedem i.e. .. note:: we'll guess, but might be wrong, and then cell0 won't work, don't say we didn't tell you
16:59:02 mriedem sdague: so in devstack, if you're not on the subnode DATABASE_HOST isn't set,
16:59:13 mriedem sdague: and on the api (primary) node, it gets defaulted to localhost: DATABASE_HOST=MYSQL_HOST=SERVICE_LOCAL_HOST=127.0.0.1
16:59:23 alaski yeah. the simple_cell_setup command seems to sit in the same place as something like devstack, i.e. it'll get you setup and show you what needs doing but probably isn't what you want for production
16:59:42 mriedem sdague: but the devstack docs say that DATABASE_HOST=$SERVICE_HOST by default if not set, and that's also what's used for the MQ host URL
16:59:49 mriedem sdague: so anything you know of that would be bad for me trying to fix that?
17:00:33 alaski mriedem: so the original reason for colocating cell0 with the api db vs the main db is that there is only ever expected to be one api db and one cell0 db, but there could be multiple "main" dbs. So thinking of where cell0 is gets confusing when there's multiple
17:00:49 openstackgerrit Sivasathurappan Radhakrishnan proposed openstack/nova: Rollback on live migration failure https://review.openstack.org/379032
17:01:02 sdague mriedem: trying to establish context
17:01:09 mriedem well i guess https://github.com/openstack-dev/devstack/blob/master/doc/source/guides/multinode-lab.rst#configure-compute-nodes says MYSQL_HOST=$SERVICE_HOST
17:01:24 mriedem sdague: in devstack today we create cell mappings on the primary node
17:01:39 mriedem sdague: and i'm seeing that the db connection in those has 127.0.0.1 as the host
17:01:50 mriedem because MYSQL_HOST=${MYSQL_HOST:-$SERVICE_LOCAL_HOST}
17:02:32 mriedem sdague: and i've got a patch up to discover the subnode compute once n-cpu is running,
17:02:46 mriedem but that's failing because the subnode can't connect to the cell db on 127.0.0.1 b/c the db doesn't exist on the subnode
17:03:19 mriedem i'm actually confused as to how the subnode nova.conf has the proper database connection url
17:03:25 sdague mriedem: which services are on the subnode?
17:03:32 mriedem n-cpu
17:03:42 mriedem oh i know why the subnode nova.conf is correct
17:03:48 sdague I'm confused, n-cpu doesn't have a db connection url
17:03:50 mriedem because d-g passes in the DATABASE_HOST in localrc
17:03:58 mriedem it does in nova.conf
17:04:03 openstackgerrit Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell0 https://review.openstack.org/421384
17:04:15 sdague mriedem: ok, but n-cpu doesn't use it, right?
17:04:16 mriedem http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/subnode-2/etc/nova/nova.conf.txt.gz
17:04:20 mriedem nope
17:04:27 sdague because that's was the whole point of conductor :)
17:04:38 mriedem yeah
17:04:46 mriedem well, my change doesn't it need it for n-cpu
17:04:57 mriedem it needs it for the discover_hosts CLI to run and map the subnode compute host to the parent cell
17:05:14 mriedem but that parent cell is created when setting up the api node, which thinks it's db host ip is 127.0.0.1
17:05:27 mriedem anywho, i think it's a simple fix, i'll just push that up and we can see
17:05:57 mriedem it's a really weird mix of d-g doesn't do something for the primary node, but does for the subnode, and you have to keep that all in your head at the same time to know what's what
17:06:28 mriedem d-g should probably just always set DATABASE_HOST in localrc for multinode jobs
17:06:32 mriedem then devstack doesn't need to guess
17:06:49 owalsh mriedem, bauzas: hola, I've a question about the cell_v2 api db sync...
17:07:19 mriedem melwitt: you can continue doing all that you droppe doing from earlier pings, i've figured out the problem :)
17:07:23 mriedem owalsh: shoot
17:07:32 mriedem owalsh: first, that's not a thing
17:07:34 mriedem but ok
17:08:00 sdague mriedem: I think the DATABASE_HOST was jammed in for cinder
17:08:04 openstackgerrit int32bit proposed openstack/python-novaclient: Add CLI to show instance usage audit logs https://review.openstack.org/409413
17:08:08 owalsh mriedem: if a user installs the N tripleo undercloud but upgrades to O before adding any ironic nodes the sync will fail
17:08:20 owalsh mriedem: here https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L51
17:08:20 mriedem N = newton?
17:08:28 owalsh ^ yes
17:08:29 sdague and because it never broken anything else, it was never looked at
17:08:47 sdague mriedem: https://review.openstack.org/#/c/106043/ is where the subnode setting of that comes from
17:09:17 mriedem sdague: yeah, i could probably have a patch in both projects
17:09:17 mriedem sdague: yeah i know
17:09:17 mriedem i was in there yesterday
17:09:39 mriedem sdague: specifically this https://review.openstack.org/#/c/106043/23/devstack-vm-gate.sh@312
17:09:59 mriedem i'm saying we should just always set DATABASE_HOST in the localrc regardless of node
17:10:30 owalsh mriedem: question being is that check absolutely necessary?
17:11:01 owalsh mriedem: if a user has not run simple setup then it will fail a previous check. If map_cell0/create_cell etc... is being used then can we assume they know what they are doing
17:11:07 mriedem owalsh: that check assumed that if you were doing a fresh install there would be no flavors yet and it would opt out
17:11:13 mriedem *noop i mean
17:11:29 mriedem but if there were flavors, it's an upgrade and there would be computes
17:11:29 owalsh mriedem: ah, forgot to mention that installing the undercloud creates default flavors
17:11:48 mriedem gdi
17:11:51 mriedem can someone please reset the 'days since tripleo disaster' counter?

Earlier   Later