| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 10:54:43 | vdrok | sdague: just guessing, in the failing job I see transport_url with cell1 at http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova_cell1.conf.txt.gz but without cell1 at | |
| 10:54:43 | vdrok | http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova.conf.txt.gz | |
| 10:56:29 | sdague | ok, we'll need dansmith I think. Exactly where the mismatches are here are beyond me. | |
| 10:59:41 | vdrok | same here :( | |
| 11:10:57 | sdague | vdrok: actually, I might have figured it out | |
| 11:11:03 | sdague | we were missing one last conditional | |
| 11:11:11 | sdague | https://review.openstack.org/487809 | |
| 11:12:36 | sdague | I changed the ironic patch to depend on that one | |
| 11:20:00 | takashin | oomichi: Are you around? | |
| 11:23:04 | vdrok | sdague: cool, thank you! | |
| 11:23:29 | sdague | vdrok: fingers crossed | |
| 11:24:09 | Ritesh | Hi, | |
| 11:24:15 | Ritesh | Can anyone please review this patch | |
| 11:24:28 | Ritesh | review pending since long time | |
| 11:24:30 | Ritesh | https://review.openstack.org/#/c/457886/ | |
| 11:32:48 | openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | |
| 11:45:17 | vdrok | sdague: ouch http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cpu.txt.gz#_Jul_27_11_32_29_018076 | |
| 11:47:10 | sdague | vdrok: ok, the conductor didn't get the right config file - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cond.txt.gz#_Jul_27_11_29_55_156482 | |
| 11:47:20 | sdague | that's solvable | |
| 11:50:53 | sdague | vdrok: ok, lets try again | |
| 11:51:32 | vdrok | yup :) | |
| 11:54:02 | openstackgerrit | Yaroslav proposed openstack/nova master: Added Overbooking DB quota driver https://review.openstack.org/484359 | |
| 12:03:24 | sean-k-mooney | o/ would people object to adding a numa aware varient of reserved_host_memory_mb in queens. we have been seeing OOM errors on our ci systems pereodically for some time even when the system has several GB of free ram. | |
| 12:04:39 | sean-k-mooney | turns out that the OOM killer kicks in when any numa node is out of memory and hte kernel need to alloacte memory there not when the system is out of memory globally | |
| 12:14:18 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 12:21:28 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 12:24:08 | catintheroof | Hi, does anyone has a good guide on how to configure live migration on ocata ? doing the same that worked as of mitaka, doesnt anymore | |
| 12:27:46 | mriedem | oh hi everybody | |
| 12:28:09 | vdrok | sdague: gah, same thing, compute does not get the build instance request http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz | |
| 12:28:15 | vdrok | morning mriedem | |
| 12:29:10 | openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | |
| 12:29:39 | mriedem | vdrok: did my attempts at dumping the rabbitmqctl report ever work? | |
| 12:30:14 | vdrok | mriedem: seems like that file is not there http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/rabbitmq/ | |
| 12:30:26 | mriedem | yeah...i was getting permission denied errors | |
| 12:31:26 | mriedem | http://logs.openstack.org/64/487664/2/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/c20a804/logs/devstack-gate-cleanup-host.txt | |
| 12:31:35 | mriedem | 2017-07-27 03:38:03.562 | /home/jenkins/workspace/gate-swift-dsvm-functional-ubuntu-xenial-nv/devstack-gate/functions.sh: line 669: /var/log/rabbitmq/cleanup-host-report.txt: Permission denied | |
| 12:31:55 | mriedem | i was thinking maybe because i couldn't write to /var/log/rabbitmq, so i moved it but then it's not even getting called | |
| 12:32:06 | mriedem | https://review.openstack.org/#/c/487664/3/functions.sh | |
| 12:32:23 | mriedem | maybe because we don't have anything in /opt/stack/logs/rabbitmq | |
| 12:33:00 | mriedem | heh, this is wrong anyway | |
| 12:33:01 | mriedem | if [ -f $BASE/logs/rabbitmq/ ]; then | |
| 12:33:04 | mriedem | -f should be -d | |
| 12:34:00 | vdrok | hrm :) | |
| 12:35:39 | sdague | vdrok: well conductor seems like it's working | |
| 12:36:33 | vdrok | is there some kind of diagram for dummies about who calls what in cellsv2 setup? | |
| 12:36:53 | mriedem | vdrok: https://review.openstack.org/#/c/487183 is a start | |
| 12:37:08 | mriedem | vdrok: with singleconductor being set, it should be the same as before the multi-conductor thing | |
| 12:37:15 | vdrok | sdague: yup, that is solved :) | |
| 12:37:20 | vdrok | mriedem: thanks! | |
| 12:37:39 | mriedem | dansmith and i were looking at it last night, and n-cpu, n-sch and n-super-cond are all using the same nova.conf with the same transport_url | |
| 12:38:01 | sdague | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz#_Jul_27_12_08_16_707963 - what's up here? | |
| 12:38:03 | mriedem | i traced the build request from super-conductor to scheduler back to super-conductor to the point that super-conductor should rpc cast to nova-compute | |
| 12:38:07 | sdague | That looks like ironic API failed | |
| 12:38:11 | mriedem | but the request never got to n-cpu | |
| 12:38:19 | mriedem | sdague: that's latent | |
| 12:38:36 | sdague | mriedem: oh... would be nice to fix that :P | |
| 12:38:39 | mriedem | sdague: vdrok: opened a bug for that yesterday https://bugs.launchpad.net/ironic/+bug/1706772 | |
| 12:38:39 | openstack | Launchpad bug 1706772 in Ironic "InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on startup with Ironic driver" [Undecided,Confirmed] | |
| 12:38:49 | mriedem | sdague: sure, but it's some keystone thing in ironic-api | |
| 12:39:05 | mriedem | endpoint discovery blows up | |
| 12:39:28 | sdague | mriedem: ok | |
| 12:39:28 | mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-ir-api.txt.gz#_Jul_27_12_08_16_701856 | |
| 12:39:58 | mriedem | so we ended last night knowing that conductor and scheduler are working but when super conductor should cast to n-cpu, everything stops there | |
| 12:40:06 | mriedem | super-conductor logs stop, rabbitmq logs stop | |
| 12:40:22 | mriedem | and the request isn't going to like the cell1 conductor, there is nothing in it's logs | |
| 12:40:52 | mriedem | so i've got the d-g patch to try and dump the rabbitmqctl report when cleaning up the host and collecting logs | |
| 12:41:08 | mriedem | but so far i haven't had a run with that work yet | |
| 12:41:17 | mriedem | https://review.openstack.org/#/c/487664/ | |
| 12:43:56 | vdrok | mriedem: sdague also uploaded this one https://review.openstack.org/487809 | |
| 12:44:28 | sdague | mriedem: the fix was definitely wrong last night, it was still spawning 2 conductors | |
| 12:47:18 | vdrok | sdague: mriedem: regarding that bug, it seems like it's the same thing again, http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/apache/error.txt.gz, apache was being restarted at that moment | |
| 12:47:57 | vdrok | we did work around it already a couple of times | |
| 12:48:15 | sdague | vdrok: why apache getting restarted? | |
| 12:49:14 | vdrok | sdague: previously it was because of some configuration changes, not sure why now, need to look into logs | |
| 12:49:31 | sdague | anyway... the main issue | |
| 12:50:03 | sdague | mriedem: so with this nova.conf - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova.conf.txt.gz | |
| 12:50:08 | mriedem | sdague: yeah the 2 conductors was the culprit, but dan and i couldn't see or think of anything about why that would be causing issues | |
| 12:50:11 | sdague | how do we know where cell0 db is? | |
| 12:50:32 | mriedem | sdague: conductor gets the cell mappings from the api db, | |
| 12:50:38 | mriedem | and the cell mapping has the cell0 mq and db urls in it | |
| 12:51:44 | mriedem | cdent: replied some and asked a question about another test in here https://review.openstack.org/#/c/487589/ | |
| 12:51:57 | cdent | yeah, just reading/responding to that. thanks | |
| 12:52:21 | mriedem | cdent: the *best* way to tell if an instance is being migrated would be to check if it has a migration_context attribute in it | |
| 12:52:36 | sdague | mriedem: ok, so which config file should the single conductor be running off of? | |
| 12:52:54 | mriedem | sdague: nova.conf | |
| 12:53:31 | sdague | which gives it a transport url of - transport_url = rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/ | |
| 12:53:31 | sdague | ||
| 12:54:13 | mriedem | cdent: i share concerns about relying on the existing allocations to know if we're doing a migration or not | |
| 12:54:26 | mriedem | because of (1) timing and (2) weird operatoins like soft delete and shelve | |
| 12:54:40 | mriedem | and the wonkiness that is the RT | |
| 12:55:19 | mriedem | sdague: so in the singleconductor case, i think the only nova config that matters is nova.conf http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/ | |
| 12:55:25 | mriedem | nova-cpu.conf and nova_cell1.conf aren't used | |
| 12:55:47 | mriedem | nova-cpu.conf wouldn't even work b/c it doesn't have any information in it about which compute driver to use, or how to talk to placement/cinder/neutron/etc | |
| 12:56:35 | mriedem | the cell1 mapping is created here: | |
| 12:56:35 | mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/devstacklog.txt.gz#_2017-07-27_12_04_56_099 | |
| 12:56:42 | mriedem | nova-manage --config-file /etc/nova/nova.conf --config-file /etc/nova/nova_cell1.conf cell_v2 create_cell --name cell1 | |
| 12:56:50 | mriedem | note that is using nova_cell1.conf rather than nova.conf | |
| 12:56:52 | sdague | mriedem: sure, where would the message queue get set | |
| 12:57:17 | sdague | maybe that's the missing piece, dumping the cell mappings | |
| 12:57:32 | mriedem | well we know the mq and db for cell1, it's taken from http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova_cell1.conf.txt.gz | |