Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-17
15:56:39 openstackgerrit Zhenyu Zheng proposed openstack/nova: Add sort_key white list for server list/detail https://review.openstack.org/415142
15:57:50 openstackgerrit Zhenyu Zheng proposed openstack/nova: Add new policy for server list/detail with all_tenants https://review.openstack.org/415330
16:08:16 mriedem tdurakov: markus_z: for https://review.openstack.org/#/c/347471/ i see it depends on a tempest patch to enable the test, do we have a patch up anywhere that depends on the nova change to see it all working and running the serial console tests in the live migration job?
16:08:43 mriedem or is it self-testing?
16:09:02 markus_z mriedem: it's self testing
16:09:31 mriedem oh really http://logs.openstack.org/71/347471/43/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/c427a53/console.html#_2017-01-17_13_10_19_246017
16:09:59 markus_z http://logs.openstack.org/71/347471/43/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/c427a53/console.html#_2017-01-17_13_10_22_840459
16:10:45 markus_z We have to skip in in grenade. Let me search for the Ocata-Ocata case.
16:11:04 mriedem oh right
16:11:17 mriedem this is the o-o one http://logs.openstack.org/71/347471/43/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/0b45362/console.html#_2017-01-17_12_34_13_290332
16:11:21 mriedem still skipped
16:11:52 mriedem melwitt: i'm struggling trying to get discover_hosts to work on the subnode in a multinode job http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/subnode-2/devstacklog.txt.gz#_2017-01-17_14_49_15_138
16:11:59 mriedem melwitt: please tell me there is something you know about this that i don't
16:12:11 mriedem the subnode nova.conf has the api_database connection and we're getting the cell mappings from the api db ok,
16:12:23 mriedem it's the target_cell that's failing on the non-cell0 cell
16:12:37 mriedem for some reason it's trying to connect to 127.0.0.1
16:13:13 markus_z mriedem: That was the first run before the hook enables the serial console. After the hook we have http://logs.openstack.org/71/347471/43/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/0b45362/console.html#_2017-01-17_12_35_19_716558
16:13:56 markus_z mriedem: The teardown of the hook sets everything back, so we have http://logs.openstack.org/71/347471/43/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/0b45362/console.html#_2017-01-17_12_36_38_329078
16:20:19 shaner hi all - any idea why live-migrations won't show up when issuing 'migrations-list'?
16:20:30 shaner (in kilo)
16:21:07 shaner only those that were migrated using 'migrate' show up .. vs 'live-migration'
16:25:31 jaypipes shaner: pls see /topic. best to ask on ask.openstack.org or mailing list for usage questions.
16:25:53 shaner jaypipes, thx, will do
16:26:23 mriedem migrate is cold migration
16:27:27 openstackgerrit int32bit proposed openstack/python-novaclient: Allow multiple tag add/delete from cli https://review.openstack.org/414635
16:28:12 shaner mriedem, shouldn't 'migration-list' list all types of migrations though?
16:28:59 mriedem i don't think until 2.23 http://docs.openstack.org/developer/nova/api_microversion_history.html#id21
16:29:23 mriedem fwiw this is the api code in kilo https://github.com/openstack/nova/blob/kilo-eol/nova/api/openstack/compute/contrib/migrations.py#L53
16:30:00 openstackgerrit int32bit proposed openstack/nova: Choose VMware datastore by the least provisioned space https://review.openstack.org/402632
16:36:36 mriedem shaner: was the migration in progress or complete? if the migration was complete i wonder if the migration record was deleted
16:36:50 mriedem at this point kilo is pretty far eol so it would take some investigating
16:37:47 shaner mriedem, it seems either way, in-progress/complete, it doesn't show up
16:37:58 shaner I've tested it several times to no avail
16:38:13 shaner appreciate the info!
16:38:46 mriedem have you looked at the migrations table in the nova db?
16:39:07 mriedem are there any records where deleted = 0?
16:39:39 shaner mriedem, let me check
16:43:13 mriedem melwitt: hmm this is interesting
16:43:28 mriedem the db connection for the cell0 mapping in this failed run has localhost in it http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/devstacklog.txt.gz#_2017-01-17_14_43_53_943
16:43:44 mriedem nova-manage cell_v2 map_cell0 --database_connection 'mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8'
16:45:15 mriedem when we call simple_cell_setup the transport_url has a valid ip in it nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@10.20.124.3:5672/
16:45:40 andreykurilin mriedem: hi! I remember you spent a bit of time investigation an issue with novaclient's gate failures (which was fixed by blocking openstacksdk 0.9.12). If you have any details please share them here - https://review.openstack.org/#/c/418663
16:47:28 mriedem andreykurilin: done
16:47:34 andreykurilin mriedem: thanks
16:50:12 mriedem hmm wtf http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/devstacklog.txt.gz#_2017-01-17_14_27_19_600
16:50:19 mriedem BASE_SQL_CONN=mysql+pymysql://root:secretmysql@127.0.0.1
16:53:18 alaski mriedem: yesterday was a holiday for me so I missed your message. Thanks for not letting a national figure distract from the important birthday
16:54:07 mriedem alaski: you're MY national figure
16:54:13 shaner mriedem, all items in migrations db are cold migrations and indeed have 'deleted = 0'
16:54:44 shaner I don't see any from the past live-migrations I've done, so it seems, a live-migration isn't updating this table?
16:54:45 alaski mriedem: :)
16:54:50 mriedem DATABASE_HOST=MYSQL_HOST=SERVICE_LOCAL_HOST=127.0.0.1 blarg!
16:54:56 alaski mriedem: I see there was a question of naming, did that get resolved?
16:55:17 mriedem shaner: would have to check if live migrations were tracked with migrations records in kilo
16:55:52 mriedem alaski: sort of https://review.openstack.org/#/c/420439/
16:55:58 mriedem there was much gnashing of teeth
16:56:12 mriedem alaski: https://review.openstack.org/#/c/420439/5/nova/cmd/manage.py@1122
16:56:37 mriedem basically we weren't sure about the original intent of using the api_database.connection for the cell0 netloc and name
16:56:53 mriedem but all agreed that nova_api_cell0 was a confusing db name for a db that is not using the api db schema
16:57:18 alaski yep
16:57:19 mriedem but also agreed that cell0 might not be on the same host as the api db, or main db,
16:57:38 mriedem and finally agreed that us trying to guess is a bad idea and people should just provide the value explicitly
16:57:45 alaski right, in a large deployment I expect all dbs to be separate
16:57:58 mriedem we probably need to make a note in our docs for ^
16:58:16 mriedem i.e. .. note:: we'll guess, but might be wrong, and then cell0 won't work, don't say we didn't tell you
16:59:02 mriedem sdague: so in devstack, if you're not on the subnode DATABASE_HOST isn't set,
16:59:13 mriedem sdague: and on the api (primary) node, it gets defaulted to localhost: DATABASE_HOST=MYSQL_HOST=SERVICE_LOCAL_HOST=127.0.0.1
16:59:23 alaski yeah. the simple_cell_setup command seems to sit in the same place as something like devstack, i.e. it'll get you setup and show you what needs doing but probably isn't what you want for production
16:59:42 mriedem sdague: but the devstack docs say that DATABASE_HOST=$SERVICE_HOST by default if not set, and that's also what's used for the MQ host URL
16:59:49 mriedem sdague: so anything you know of that would be bad for me trying to fix that?
17:00:33 alaski mriedem: so the original reason for colocating cell0 with the api db vs the main db is that there is only ever expected to be one api db and one cell0 db, but there could be multiple "main" dbs. So thinking of where cell0 is gets confusing when there's multiple
17:00:49 openstackgerrit Sivasathurappan Radhakrishnan proposed openstack/nova: Rollback on live migration failure https://review.openstack.org/379032
17:01:02 sdague mriedem: trying to establish context
17:01:09 mriedem well i guess https://github.com/openstack-dev/devstack/blob/master/doc/source/guides/multinode-lab.rst#configure-compute-nodes says MYSQL_HOST=$SERVICE_HOST
17:01:24 mriedem sdague: in devstack today we create cell mappings on the primary node
17:01:39 mriedem sdague: and i'm seeing that the db connection in those has 127.0.0.1 as the host
17:01:50 mriedem because MYSQL_HOST=${MYSQL_HOST:-$SERVICE_LOCAL_HOST}
17:02:32 mriedem sdague: and i've got a patch up to discover the subnode compute once n-cpu is running,
17:02:46 mriedem but that's failing because the subnode can't connect to the cell db on 127.0.0.1 b/c the db doesn't exist on the subnode
17:03:19 mriedem i'm actually confused as to how the subnode nova.conf has the proper database connection url
17:03:25 sdague mriedem: which services are on the subnode?
17:03:32 mriedem n-cpu
17:03:42 mriedem oh i know why the subnode nova.conf is correct
17:03:48 sdague I'm confused, n-cpu doesn't have a db connection url
17:03:50 mriedem because d-g passes in the DATABASE_HOST in localrc
17:03:58 mriedem it does in nova.conf
17:04:03 openstackgerrit Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell0 https://review.openstack.org/421384
17:04:15 sdague mriedem: ok, but n-cpu doesn't use it, right?
17:04:16 mriedem http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/subnode-2/etc/nova/nova.conf.txt.gz
17:04:20 mriedem nope
17:04:27 sdague because that's was the whole point of conductor :)
17:04:38 mriedem yeah
17:04:46 mriedem well, my change doesn't it need it for n-cpu
17:04:57 mriedem it needs it for the discover_hosts CLI to run and map the subnode compute host to the parent cell
17:05:14 mriedem but that parent cell is created when setting up the api node, which thinks it's db host ip is 127.0.0.1
17:05:27 mriedem anywho, i think it's a simple fix, i'll just push that up and we can see
17:05:57 mriedem it's a really weird mix of d-g doesn't do something for the primary node, but does for the subnode, and you have to keep that all in your head at the same time to know what's what
17:06:28 mriedem d-g should probably just always set DATABASE_HOST in localrc for multinode jobs
17:06:32 mriedem then devstack doesn't need to guess
17:06:49 owalsh mriedem, bauzas: hola, I've a question about the cell_v2 api db sync...

Earlier   Later