| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 12:28:09 | vdrok | sdague: gah, same thing, compute does not get the build instance request http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz | |
| 12:28:15 | vdrok | morning mriedem | |
| 12:29:10 | openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | |
| 12:29:39 | mriedem | vdrok: did my attempts at dumping the rabbitmqctl report ever work? | |
| 12:30:14 | vdrok | mriedem: seems like that file is not there http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/rabbitmq/ | |
| 12:30:26 | mriedem | yeah...i was getting permission denied errors | |
| 12:31:26 | mriedem | http://logs.openstack.org/64/487664/2/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/c20a804/logs/devstack-gate-cleanup-host.txt | |
| 12:31:35 | mriedem | 2017-07-27 03:38:03.562 | /home/jenkins/workspace/gate-swift-dsvm-functional-ubuntu-xenial-nv/devstack-gate/functions.sh: line 669: /var/log/rabbitmq/cleanup-host-report.txt: Permission denied | |
| 12:31:55 | mriedem | i was thinking maybe because i couldn't write to /var/log/rabbitmq, so i moved it but then it's not even getting called | |
| 12:32:06 | mriedem | https://review.openstack.org/#/c/487664/3/functions.sh | |
| 12:32:23 | mriedem | maybe because we don't have anything in /opt/stack/logs/rabbitmq | |
| 12:33:00 | mriedem | heh, this is wrong anyway | |
| 12:33:01 | mriedem | if [ -f $BASE/logs/rabbitmq/ ]; then | |
| 12:33:04 | mriedem | -f should be -d | |
| 12:34:00 | vdrok | hrm :) | |
| 12:35:39 | sdague | vdrok: well conductor seems like it's working | |
| 12:36:33 | vdrok | is there some kind of diagram for dummies about who calls what in cellsv2 setup? | |
| 12:36:53 | mriedem | vdrok: https://review.openstack.org/#/c/487183 is a start | |
| 12:37:08 | mriedem | vdrok: with singleconductor being set, it should be the same as before the multi-conductor thing | |
| 12:37:15 | vdrok | sdague: yup, that is solved :) | |
| 12:37:20 | vdrok | mriedem: thanks! | |
| 12:37:39 | mriedem | dansmith and i were looking at it last night, and n-cpu, n-sch and n-super-cond are all using the same nova.conf with the same transport_url | |
| 12:38:01 | sdague | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz#_Jul_27_12_08_16_707963 - what's up here? | |
| 12:38:03 | mriedem | i traced the build request from super-conductor to scheduler back to super-conductor to the point that super-conductor should rpc cast to nova-compute | |
| 12:38:07 | sdague | That looks like ironic API failed | |
| 12:38:11 | mriedem | but the request never got to n-cpu | |
| 12:38:19 | mriedem | sdague: that's latent | |
| 12:38:36 | sdague | mriedem: oh... would be nice to fix that :P | |
| 12:38:39 | mriedem | sdague: vdrok: opened a bug for that yesterday https://bugs.launchpad.net/ironic/+bug/1706772 | |
| 12:38:39 | openstack | Launchpad bug 1706772 in Ironic "InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on startup with Ironic driver" [Undecided,Confirmed] | |
| 12:38:49 | mriedem | sdague: sure, but it's some keystone thing in ironic-api | |
| 12:39:05 | mriedem | endpoint discovery blows up | |
| 12:39:28 | sdague | mriedem: ok | |
| 12:39:28 | mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-ir-api.txt.gz#_Jul_27_12_08_16_701856 | |
| 12:39:58 | mriedem | so we ended last night knowing that conductor and scheduler are working but when super conductor should cast to n-cpu, everything stops there | |
| 12:40:06 | mriedem | super-conductor logs stop, rabbitmq logs stop | |
| 12:40:22 | mriedem | and the request isn't going to like the cell1 conductor, there is nothing in it's logs | |
| 12:40:52 | mriedem | so i've got the d-g patch to try and dump the rabbitmqctl report when cleaning up the host and collecting logs | |
| 12:41:08 | mriedem | but so far i haven't had a run with that work yet | |
| 12:41:17 | mriedem | https://review.openstack.org/#/c/487664/ | |
| 12:43:56 | vdrok | mriedem: sdague also uploaded this one https://review.openstack.org/487809 | |
| 12:44:28 | sdague | mriedem: the fix was definitely wrong last night, it was still spawning 2 conductors | |
| 12:47:18 | vdrok | sdague: mriedem: regarding that bug, it seems like it's the same thing again, http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/apache/error.txt.gz, apache was being restarted at that moment | |
| 12:47:57 | vdrok | we did work around it already a couple of times | |
| 12:48:15 | sdague | vdrok: why apache getting restarted? | |
| 12:49:14 | vdrok | sdague: previously it was because of some configuration changes, not sure why now, need to look into logs | |
| 12:49:31 | sdague | anyway... the main issue | |
| 12:50:03 | sdague | mriedem: so with this nova.conf - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova.conf.txt.gz | |
| 12:50:08 | mriedem | sdague: yeah the 2 conductors was the culprit, but dan and i couldn't see or think of anything about why that would be causing issues | |
| 12:50:11 | sdague | how do we know where cell0 db is? | |
| 12:50:32 | mriedem | sdague: conductor gets the cell mappings from the api db, | |
| 12:50:38 | mriedem | and the cell mapping has the cell0 mq and db urls in it | |
| 12:51:44 | mriedem | cdent: replied some and asked a question about another test in here https://review.openstack.org/#/c/487589/ | |
| 12:51:57 | cdent | yeah, just reading/responding to that. thanks | |
| 12:52:21 | mriedem | cdent: the *best* way to tell if an instance is being migrated would be to check if it has a migration_context attribute in it | |
| 12:52:36 | sdague | mriedem: ok, so which config file should the single conductor be running off of? | |
| 12:52:54 | mriedem | sdague: nova.conf | |
| 12:53:31 | sdague | which gives it a transport url of - transport_url = rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/ | |
| 12:53:31 | sdague | ||
| 12:54:13 | mriedem | cdent: i share concerns about relying on the existing allocations to know if we're doing a migration or not | |
| 12:54:26 | mriedem | because of (1) timing and (2) weird operatoins like soft delete and shelve | |
| 12:54:40 | mriedem | and the wonkiness that is the RT | |
| 12:55:19 | mriedem | sdague: so in the singleconductor case, i think the only nova config that matters is nova.conf http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/ | |
| 12:55:25 | mriedem | nova-cpu.conf and nova_cell1.conf aren't used | |
| 12:55:47 | mriedem | nova-cpu.conf wouldn't even work b/c it doesn't have any information in it about which compute driver to use, or how to talk to placement/cinder/neutron/etc | |
| 12:56:35 | mriedem | the cell1 mapping is created here: | |
| 12:56:35 | mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/devstacklog.txt.gz#_2017-07-27_12_04_56_099 | |
| 12:56:42 | mriedem | nova-manage --config-file /etc/nova/nova.conf --config-file /etc/nova/nova_cell1.conf cell_v2 create_cell --name cell1 | |
| 12:56:50 | mriedem | note that is using nova_cell1.conf rather than nova.conf | |
| 12:56:52 | sdague | mriedem: sure, where would the message queue get set | |
| 12:57:17 | sdague | maybe that's the missing piece, dumping the cell mappings | |
| 12:57:32 | mriedem | well we know the mq and db for cell1, it's taken from http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 12:57:43 | sdague | right | |
| 12:57:50 | sdague | but that's not used anywhere | |
| 12:58:01 | mriedem | it's used when creating the cell1 mapping | |
| 12:58:02 | sdague | so that's stating that the cell1 mq is going to be on a vhost | |
| 12:58:06 | mriedem | nova-manage --config-file /etc/nova/nova.conf --config-file /etc/nova/nova_cell1.conf cell_v2 create_cell --name cell1 | |
| 12:58:13 | sdague | but nova-compute is started not listening to that vhost | |
| 12:59:32 | mriedem | right nova-compute is listening on rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/ | |
| 12:59:42 | mriedem | the cell1 mapping is sending to rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/nova_cell1 | |
| 12:59:48 | sdague | but conductor isn't sending messages there, right? | |
| 13:00:09 | mriedem | because https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1042 | |
| 13:00:21 | sdague | https://github.com/openstack-dev/devstack/blob/9596fdddccd04c26aa5adb923b9bd8e64c6593ec/lib/nova#L709-L712 | |
| 13:00:21 | mriedem | conductor does an mq switch when it casts to compute | |
| 13:01:02 | sdague | mriedem: ok, so are you agreeing or disagreeing with me that nova-cond and nova-compute aren't talking on the same mq :) | |
| 13:01:37 | mriedem | can i phone a friend? | |
| 13:01:50 | mriedem | i think i'm agreeing, | |
| 13:01:57 | mriedem | and that would explain why we're not seeing the message get to n-cpu | |
| 13:02:26 | sdague | yeh | |
| 13:02:37 | sdague | is there a nova-manage command to dump the cell mappings? | |
| 13:02:46 | sdague | I think that's kind of critical to see the mismatch | |
| 13:03:48 | mriedem | yes, | |
| 13:03:52 | mriedem | nova-manage cell_v2 list_cells | |
| 13:04:17 | mriedem | with --verbose | |
| 13:04:28 | mriedem | --verbose dumps the db and mq urls | |
| 13:05:25 | mriedem | fwiw, this is a run before the fleetify patch where we create cell1 | |
| 13:05:26 | mriedem | http://logs.openstack.org/23/485823/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/206b79e/logs/devstacklog.txt.gz#_2017-07-21_01_52_17_794 | |
| 13:05:48 | mriedem | nova-manage cell_v2 create_cell --transport-url rabbit://stackrabbit:secretrabbit@10.0.1.31:5672/ --name cell1 | |
| 13:05:51 | mriedem | there is a single nova.conf | |
| 13:06:13 | mriedem | and it's using the same transport_url http://logs.openstack.org/23/485823/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/206b79e/logs/etc/nova/nova.conf.txt.gz | |