| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-17 | |||
| 17:03:48 | sdague | I'm confused, n-cpu doesn't have a db connection url | |
| 17:03:50 | mriedem | because d-g passes in the DATABASE_HOST in localrc | |
| 17:03:58 | mriedem | it does in nova.conf | |
| 17:04:03 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_cell0 https://review.openstack.org/421384 | |
| 17:04:15 | sdague | mriedem: ok, but n-cpu doesn't use it, right? | |
| 17:04:16 | mriedem | http://logs.openstack.org/81/420981/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/6767a3a/logs/subnode-2/etc/nova/nova.conf.txt.gz | |
| 17:04:20 | mriedem | nope | |
| 17:04:27 | sdague | because that's was the whole point of conductor :) | |
| 17:04:38 | mriedem | yeah | |
| 17:04:46 | mriedem | well, my change doesn't it need it for n-cpu | |
| 17:04:57 | mriedem | it needs it for the discover_hosts CLI to run and map the subnode compute host to the parent cell | |
| 17:05:14 | mriedem | but that parent cell is created when setting up the api node, which thinks it's db host ip is 127.0.0.1 | |
| 17:05:27 | mriedem | anywho, i think it's a simple fix, i'll just push that up and we can see | |
| 17:05:57 | mriedem | it's a really weird mix of d-g doesn't do something for the primary node, but does for the subnode, and you have to keep that all in your head at the same time to know what's what | |
| 17:06:28 | mriedem | d-g should probably just always set DATABASE_HOST in localrc for multinode jobs | |
| 17:06:32 | mriedem | then devstack doesn't need to guess | |
| 17:06:49 | owalsh | mriedem, bauzas: hola, I've a question about the cell_v2 api db sync... | |
| 17:07:19 | mriedem | melwitt: you can continue doing all that you droppe doing from earlier pings, i've figured out the problem :) | |
| 17:07:23 | mriedem | owalsh: shoot | |
| 17:07:32 | mriedem | owalsh: first, that's not a thing | |
| 17:07:34 | mriedem | but ok | |
| 17:08:00 | sdague | mriedem: I think the DATABASE_HOST was jammed in for cinder | |
| 17:08:04 | openstackgerrit | int32bit proposed openstack/python-novaclient: Add CLI to show instance usage audit logs https://review.openstack.org/409413 | |
| 17:08:08 | owalsh | mriedem: if a user installs the N tripleo undercloud but upgrades to O before adding any ironic nodes the sync will fail | |
| 17:08:20 | owalsh | mriedem: here https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L51 | |
| 17:08:20 | mriedem | N = newton? | |
| 17:08:28 | owalsh | ^ yes | |
| 17:08:29 | sdague | and because it never broken anything else, it was never looked at | |
| 17:08:47 | sdague | mriedem: https://review.openstack.org/#/c/106043/ is where the subnode setting of that comes from | |
| 17:09:17 | mriedem | sdague: yeah, i could probably have a patch in both projects | |
| 17:09:17 | mriedem | sdague: yeah i know | |
| 17:09:17 | mriedem | i was in there yesterday | |
| 17:09:39 | mriedem | sdague: specifically this https://review.openstack.org/#/c/106043/23/devstack-vm-gate.sh@312 | |
| 17:09:59 | mriedem | i'm saying we should just always set DATABASE_HOST in the localrc regardless of node | |
| 17:10:30 | owalsh | mriedem: question being is that check absolutely necessary? | |
| 17:11:01 | owalsh | mriedem: if a user has not run simple setup then it will fail a previous check. If map_cell0/create_cell etc... is being used then can we assume they know what they are doing | |
| 17:11:07 | mriedem | owalsh: that check assumed that if you were doing a fresh install there would be no flavors yet and it would opt out | |
| 17:11:13 | mriedem | *noop i mean | |
| 17:11:29 | mriedem | but if there were flavors, it's an upgrade and there would be computes | |
| 17:11:29 | owalsh | mriedem: ah, forgot to mention that installing the undercloud creates default flavors | |
| 17:11:48 | mriedem | gdi | |
| 17:11:51 | mriedem | can someone please reset the 'days since tripleo disaster' counter? | |
| 17:12:48 | owalsh | mriedem: :-) | |
| 17:12:52 | mriedem | so what are your steps again? | |
| 17:13:07 | mriedem | install newton, sync api db, sync nova db, create flavors,... | |
| 17:13:16 | sdague | mriedem: https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L468-L469 - probably that should be conditional behind n-api / n-cond / n-sched enabled | |
| 17:17:03 | owalsh | mriedem: install newton, create flavors (implicitly by installing TripleO undercloud), upgrade to ocata, create cell0 db, map cell0, create cell1, sync nova db, discover_hosts (none yet), sync api db ... deploy computes, discover_hosts | |
| 17:17:26 | owalsh | mriedem: I err think... | |
| 17:17:47 | openstackgerrit | Dan Peschman proposed openstack/nova: nova-manage cell_v2 map_cell0 exit 0 https://review.openstack.org/420132 | |
| 17:18:28 | mriedem | owalsh: so between "discover_hosts (none yet), sync api db" you blow up | |
| 17:18:34 | owalsh | mriedem: yes | |
| 17:19:28 | mriedem | fudg | |
| 17:19:32 | mriedem | *e | |
| 17:20:38 | mriedem | see how nova-status handles this exact same thing https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L153-L171 | |
| 17:20:42 | mriedem | it doesn't consider it a failure | |
| 17:22:15 | owalsh | mriedem: yea, I'm thinking the db migraion could be relaxed. It will fail on the first check if the user hasn't run simple_cell_setup | |
| 17:23:24 | owalsh | mriedem: if it gets as far as the host mapping check it implies that map_cell0 and create_cell were used, or simple_cell_setup failed... | |
| 17:28:27 | mriedem | i need to reset my brain on how we can even add cell mappings before running the api db sync | |
| 17:29:53 | mriedem | in devstack we create the nova_api db and then sync it here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L694 | |
| 17:30:11 | mriedem | then later we run simple_cell_setup here https://github.com/openstack-dev/devstack/blob/3c8ccecb1f0dff855e8043ba681d1749645de738/lib/nova#L950 | |
| 17:30:28 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying experimental jobs https://review.openstack.org/421209 | |
| 17:30:28 | mriedem | oh devstack works because when we sync the api_db we don't have any flavors yet | |
| 17:30:32 | mriedem | so it noops | |
| 17:31:38 | mriedem | ok let me wrap up this thing i'm fixing first, then i can come back to this | |
| 17:34:10 | gibi | mriedem: hi! I saw you patch chasing the race in the notification sample test https://review.openstack.org/#/c/421221 | |
| 17:34:25 | mriedem | yeah | |
| 17:34:27 | gibi | mriedem: do we want to merge that to see more logs or just use to locally reproduce the problem? | |
| 17:34:35 | mriedem | gibi: more logs | |
| 17:34:42 | mriedem | and so i can fingerprint the failure in elastic-recheck | |
| 17:35:04 | gibi | mriedem: OK, then I'm totally +1 on that | |
| 17:40:10 | openstackgerrit | Diana Clarke proposed openstack/nova: nova-manage docs: cell_v2 map_instances https://review.openstack.org/421404 | |
| 17:50:56 | melwitt | mriedem: sweet! glad I could help by not being around | |
| 17:51:25 | mriedem | melwitt: i think this is the culmination of 2 days of frustration https://review.openstack.org/#/c/421399/1 | |
| 17:51:37 | mriedem | and about 4 hours of sleep | |
| 17:52:00 | dims | mriedem : wow one liner! | |
| 17:52:07 | melwitt | oof | |
| 17:53:42 | mriedem | so in devstack runs we create flavors basically last | |
| 17:53:54 | mriedem | we create the dbs and sync them, | |
| 17:53:58 | mriedem | we run simple_cell_setup | |
| 17:54:01 | mriedem | then we create the flavors | |
| 17:54:34 | mriedem | and that's how devstack avoids failing the api db migration for there being no host mappings (well we'd have host mappings after running simple_cell_setup) | |
| 17:55:18 | owalsh | ... and the api db doesn't know about hosts etc... so the flavor is the trigger yea | |
| 17:55:22 | mriedem | in any other fresh install, you can't create flavors in the api db until you sync the api db | |
| 17:55:57 | owalsh | sequence above is an upgrade, so we have the tables from newton I expect | |
| 17:56:11 | mriedem | so, we have a particularly terrible scenario where you're actively going out of your way to create the flavors in the api db in newton, then upgrade to ocata and sync the api db before having any hosts | |
| 17:56:24 | mriedem | owalsh: wtf does tripleo do this? | |
| 17:56:40 | owalsh | mriedem: it doesn't, but it's a possible sequence of event | |
| 17:56:48 | owalsh | mriedem: user would need to do this | |
| 17:57:05 | mriedem | i thought the point of tripleo was to keep the user from hurting themselves? | |
| 17:57:20 | mriedem | s/user/operator/ | |
| 17:58:21 | mriedem | i mean i guess it's valid to create the newton control plane + flavors, and then decide, i might as well upgrade this to ocata even though i don't have any computes yet | |
| 17:58:25 | mriedem | it just seems really weird | |
| 17:59:20 | owalsh | it wouldn't be typical, but if an operator did this they wouldn't be able to recover easily I expect | |
| 18:01:09 | mriedem | well, they'd have to delete the flavors | |
| 18:01:43 | mriedem | there would be 2 outs: | |
| 18:02:06 | mriedem | 1. delete the flavors, sync the api db, do the cell setup stuff, then add the flavors | |
| 18:02:32 | mriedem | 2. create the computes, run discover_hosts or simple_cell_setup, then sync the api db | |
| 18:04:35 | owalsh | should clarify, I'm talking about the all-in-one cloud that deploys the real cloud | |
| 18:06:13 | owalsh | operator could install it, maybe create custom flavors etc... upgrade to ocata, then add hosts to deploy to | |
| 18:06:26 | owalsh | far less likely to have a production cloud with no hosts | |