| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 10:39:37 | kiennt | sdague, andreaf: So, after upgrade, if I run some tests (ex.tempest), it will run in services (2 versions). That will test these mixed services could talk to each other or not. Is this right? | |
| 10:39:58 | andreaf | kiennt: yeah that's the idea | |
| 10:40:35 | openstackgerrit | Chris Dent proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | |
| 10:42:06 | kiennt | andreaf, sdague: Thank you, guys. This is very important information with me. I have stucked at this for a long time, because i'm completely new with all these things. | |
| 10:42:53 | kiennt | Seem like I only need some tempest tests after upgrade now because I already have multinode gate. | |
| 10:43:26 | openstackgerrit | Artem Tiumentcev proposed openstack/python-novaclient master: Add 'local_gb' disk quota support https://review.openstack.org/484391 | |
| 10:43:46 | openstackgerrit | Chris Dent proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 10:48:39 | sdague | andreaf: yes, primary node is installed, subnode is installed. Tests are run. Then primary node is upgraded. | |
| 10:49:43 | andreaf | sdague: oh I see, so we don't run any test after upgrade? on check resources? | |
| 10:49:47 | andreaf | s/on/only | |
| 10:50:02 | sdague | andreaf: sorry, no we run tests again at the end | |
| 10:50:25 | andreaf | sdague: ok thanks | |
| 10:50:41 | sdague | so, because this is an intergration between devstack-gate and grenade, I don't know that it's ever gotten documented very well | |
| 10:50:58 | sdague | I'd be happy to write up a webpage with what it is, if you figure out where you'd like it to live :) | |
| 10:51:28 | sdague | do we currently have overarching qa docs outside of tempest / grenade / devstack ones? | |
| 10:54:43 | vdrok | http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova.conf.txt.gz | |
| 10:54:43 | vdrok | sdague: just guessing, in the failing job I see transport_url with cell1 at http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova_cell1.conf.txt.gz but without cell1 at | |
| 10:56:29 | sdague | ok, we'll need dansmith I think. Exactly where the mismatches are here are beyond me. | |
| 10:59:41 | vdrok | same here :( | |
| 11:10:57 | sdague | vdrok: actually, I might have figured it out | |
| 11:11:03 | sdague | we were missing one last conditional | |
| 11:11:11 | sdague | https://review.openstack.org/487809 | |
| 11:12:36 | sdague | I changed the ironic patch to depend on that one | |
| 11:20:00 | takashin | oomichi: Are you around? | |
| 11:23:04 | vdrok | sdague: cool, thank you! | |
| 11:23:29 | sdague | vdrok: fingers crossed | |
| 11:24:09 | Ritesh | Hi, | |
| 11:24:15 | Ritesh | Can anyone please review this patch | |
| 11:24:28 | Ritesh | review pending since long time | |
| 11:24:30 | Ritesh | https://review.openstack.org/#/c/457886/ | |
| 11:32:48 | openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | |
| 11:45:17 | vdrok | sdague: ouch http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cpu.txt.gz#_Jul_27_11_32_29_018076 | |
| 11:47:10 | sdague | vdrok: ok, the conductor didn't get the right config file - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cond.txt.gz#_Jul_27_11_29_55_156482 | |
| 11:47:20 | sdague | that's solvable | |
| 11:50:53 | sdague | vdrok: ok, lets try again | |
| 11:51:32 | vdrok | yup :) | |
| 11:54:02 | openstackgerrit | Yaroslav proposed openstack/nova master: Added Overbooking DB quota driver https://review.openstack.org/484359 | |
| 12:03:24 | sean-k-mooney | o/ would people object to adding a numa aware varient of reserved_host_memory_mb in queens. we have been seeing OOM errors on our ci systems pereodically for some time even when the system has several GB of free ram. | |
| 12:04:39 | sean-k-mooney | turns out that the OOM killer kicks in when any numa node is out of memory and hte kernel need to alloacte memory there not when the system is out of memory globally | |
| 12:14:18 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 12:21:28 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 12:24:08 | catintheroof | Hi, does anyone has a good guide on how to configure live migration on ocata ? doing the same that worked as of mitaka, doesnt anymore | |
| 12:27:46 | mriedem | oh hi everybody | |
| 12:28:09 | vdrok | sdague: gah, same thing, compute does not get the build instance request http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz | |
| 12:28:15 | vdrok | morning mriedem | |
| 12:29:10 | openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | |
| 12:29:39 | mriedem | vdrok: did my attempts at dumping the rabbitmqctl report ever work? | |
| 12:30:14 | vdrok | mriedem: seems like that file is not there http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/rabbitmq/ | |
| 12:30:26 | mriedem | yeah...i was getting permission denied errors | |
| 12:31:26 | mriedem | http://logs.openstack.org/64/487664/2/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/c20a804/logs/devstack-gate-cleanup-host.txt | |
| 12:31:35 | mriedem | 2017-07-27 03:38:03.562 | /home/jenkins/workspace/gate-swift-dsvm-functional-ubuntu-xenial-nv/devstack-gate/functions.sh: line 669: /var/log/rabbitmq/cleanup-host-report.txt: Permission denied | |
| 12:31:55 | mriedem | i was thinking maybe because i couldn't write to /var/log/rabbitmq, so i moved it but then it's not even getting called | |
| 12:32:06 | mriedem | https://review.openstack.org/#/c/487664/3/functions.sh | |
| 12:32:23 | mriedem | maybe because we don't have anything in /opt/stack/logs/rabbitmq | |
| 12:33:00 | mriedem | heh, this is wrong anyway | |
| 12:33:01 | mriedem | if [ -f $BASE/logs/rabbitmq/ ]; then | |
| 12:33:04 | mriedem | -f should be -d | |
| 12:34:00 | vdrok | hrm :) | |
| 12:35:39 | sdague | vdrok: well conductor seems like it's working | |
| 12:36:33 | vdrok | is there some kind of diagram for dummies about who calls what in cellsv2 setup? | |
| 12:36:53 | mriedem | vdrok: https://review.openstack.org/#/c/487183 is a start | |
| 12:37:08 | mriedem | vdrok: with singleconductor being set, it should be the same as before the multi-conductor thing | |
| 12:37:15 | vdrok | sdague: yup, that is solved :) | |
| 12:37:20 | vdrok | mriedem: thanks! | |
| 12:37:39 | mriedem | dansmith and i were looking at it last night, and n-cpu, n-sch and n-super-cond are all using the same nova.conf with the same transport_url | |
| 12:38:01 | sdague | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz#_Jul_27_12_08_16_707963 - what's up here? | |
| 12:38:03 | mriedem | i traced the build request from super-conductor to scheduler back to super-conductor to the point that super-conductor should rpc cast to nova-compute | |
| 12:38:07 | sdague | That looks like ironic API failed | |
| 12:38:11 | mriedem | but the request never got to n-cpu | |
| 12:38:19 | mriedem | sdague: that's latent | |
| 12:38:36 | sdague | mriedem: oh... would be nice to fix that :P | |
| 12:38:39 | openstack | Launchpad bug 1706772 in Ironic "InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on startup with Ironic driver" [Undecided,Confirmed] | |
| 12:38:39 | mriedem | sdague: vdrok: opened a bug for that yesterday https://bugs.launchpad.net/ironic/+bug/1706772 | |
| 12:38:49 | mriedem | sdague: sure, but it's some keystone thing in ironic-api | |
| 12:39:05 | mriedem | endpoint discovery blows up | |
| 12:39:28 | mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-ir-api.txt.gz#_Jul_27_12_08_16_701856 | |
| 12:39:28 | sdague | mriedem: ok | |
| 12:39:58 | mriedem | so we ended last night knowing that conductor and scheduler are working but when super conductor should cast to n-cpu, everything stops there | |
| 12:40:06 | mriedem | super-conductor logs stop, rabbitmq logs stop | |
| 12:40:22 | mriedem | and the request isn't going to like the cell1 conductor, there is nothing in it's logs | |
| 12:40:52 | mriedem | so i've got the d-g patch to try and dump the rabbitmqctl report when cleaning up the host and collecting logs | |
| 12:41:08 | mriedem | but so far i haven't had a run with that work yet | |
| 12:41:17 | mriedem | https://review.openstack.org/#/c/487664/ | |
| 12:43:56 | vdrok | mriedem: sdague also uploaded this one https://review.openstack.org/487809 | |
| 12:44:28 | sdague | mriedem: the fix was definitely wrong last night, it was still spawning 2 conductors | |
| 12:47:18 | vdrok | sdague: mriedem: regarding that bug, it seems like it's the same thing again, http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/apache/error.txt.gz, apache was being restarted at that moment | |
| 12:47:57 | vdrok | we did work around it already a couple of times | |
| 12:48:15 | sdague | vdrok: why apache getting restarted? | |
| 12:49:14 | vdrok | sdague: previously it was because of some configuration changes, not sure why now, need to look into logs | |
| 12:49:31 | sdague | anyway... the main issue | |
| 12:50:03 | sdague | mriedem: so with this nova.conf - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova.conf.txt.gz | |
| 12:50:08 | mriedem | sdague: yeah the 2 conductors was the culprit, but dan and i couldn't see or think of anything about why that would be causing issues | |
| 12:50:11 | sdague | how do we know where cell0 db is? | |
| 12:50:32 | mriedem | sdague: conductor gets the cell mappings from the api db, | |
| 12:50:38 | mriedem | and the cell mapping has the cell0 mq and db urls in it | |
| 12:51:44 | mriedem | cdent: replied some and asked a question about another test in here https://review.openstack.org/#/c/487589/ | |
| 12:51:57 | cdent | yeah, just reading/responding to that. thanks | |
| 12:52:21 | mriedem | cdent: the *best* way to tell if an instance is being migrated would be to check if it has a migration_context attribute in it | |
| 12:52:36 | sdague | mriedem: ok, so which config file should the single conductor be running off of? | |
| 12:52:54 | mriedem | sdague: nova.conf | |