| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 09:31:20 | bauzas | and we only have like 6 hours for that | |
| 09:31:55 | bauzas | I totally understand you would be sad if we would push it to Queens, but the main problem we had was about not a lot of reviews | |
| 09:32:15 | bauzas | given we'll open Queens in two weeks, I do wonder if we could just push it for 2 weeks | |
| 09:32:28 | bauzas | the main problem is about having the feature not in Pike | |
| 10:02:46 | sdague | vdrok: how is the ironic patch going? | |
| 10:03:25 | vdrok | sdague: I have not yet had a chance to look, will do in ~half an hour | |
| 10:03:32 | alex_xu | bauzas: why we don't have API for delete the aggregates in the placement? I remember there are some discussion, but I forget the detail... | |
| 10:06:12 | sdague | johnthetubaguy: https://review.openstack.org/#/c/357726 - this patch is ready to go if we reno it | |
| 10:06:18 | sdague | can you rev for that? | |
| 10:30:28 | kiennt | sdague: Hi, i am working on Heat gate job for rolling upgrade testing. I already setup multinode grenade gate job but it doesn't have any tests for testing mixed version. | |
| 10:31:02 | kiennt | I find out that Nova already had rolling-upgrade tag, so I want to ask about the way Nova do the tests | |
| 10:31:08 | kiennt | thank you | |
| 10:32:53 | openstackgerrit | Sean Dague proposed openstack/nova master: Assume neutron port_binding extensions enabled https://review.openstack.org/357726 | |
| 10:32:54 | openstackgerrit | Sean Dague proposed openstack/nova master: Use admin context to create neutron ports https://review.openstack.org/357540 | |
| 10:33:44 | sdague | kiennt: it's all about what makes sense for heat and it's collection of services. How would you expect heat would be upgraded across a set of nodes during upgrade? | |
| 10:34:13 | sdague | johnthetubaguy - ok, reno added for - https://review.openstack.org/#/c/357726/ | |
| 10:35:38 | andreaf | sdague: around? is there a flag in d-g or grenade that decides which host is upgraded in a multinode setup | |
| 10:36:05 | sdague | andreaf: no, it's baked in logic | |
| 10:36:13 | sdague | the primary node is what is upgraded | |
| 10:36:20 | sdague | subnodes are not | |
| 10:37:07 | andreaf | sdague: ok that was my impression looking at devstack-vm-gate.sh, thanks | |
| 10:37:39 | andreaf | kiennt: ^^^ | |
| 10:37:58 | andreaf | sdague: so everything is setup on old first and then primary is upgraded to new right? | |
| 10:38:01 | vdrok | sdague: so as mriedem said, in that job I see that the conductor called into scheduler which selected a host http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/screen-n-sch.txt.gz#_Jul_27_04_27_52_998156, then conductor proceeded (as I see | |
| 10:38:01 | vdrok | http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/screen-n-super-cond.txt.gz#_Jul_27_04_27_53_252817, which happens after select_destinations), but in the compute log there is no build_and_run_instance, don't see the 'Starting instnace...' | |
| 10:39:37 | kiennt | sdague, andreaf: So, after upgrade, if I run some tests (ex.tempest), it will run in services (2 versions). That will test these mixed services could talk to each other or not. Is this right? | |
| 10:39:58 | andreaf | kiennt: yeah that's the idea | |
| 10:40:35 | openstackgerrit | Chris Dent proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589 | |
| 10:42:06 | kiennt | andreaf, sdague: Thank you, guys. This is very important information with me. I have stucked at this for a long time, because i'm completely new with all these things. | |
| 10:42:53 | kiennt | Seem like I only need some tempest tests after upgrade now because I already have multinode gate. | |
| 10:43:26 | openstackgerrit | Artem Tiumentcev proposed openstack/python-novaclient master: Add 'local_gb' disk quota support https://review.openstack.org/484391 | |
| 10:43:46 | openstackgerrit | Chris Dent proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566 | |
| 10:48:39 | sdague | andreaf: yes, primary node is installed, subnode is installed. Tests are run. Then primary node is upgraded. | |
| 10:49:43 | andreaf | sdague: oh I see, so we don't run any test after upgrade? on check resources? | |
| 10:49:47 | andreaf | s/on/only | |
| 10:50:02 | sdague | andreaf: sorry, no we run tests again at the end | |
| 10:50:25 | andreaf | sdague: ok thanks | |
| 10:50:41 | sdague | so, because this is an intergration between devstack-gate and grenade, I don't know that it's ever gotten documented very well | |
| 10:50:58 | sdague | I'd be happy to write up a webpage with what it is, if you figure out where you'd like it to live :) | |
| 10:51:28 | sdague | do we currently have overarching qa docs outside of tempest / grenade / devstack ones? | |
| 10:54:43 | vdrok | sdague: just guessing, in the failing job I see transport_url with cell1 at http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova_cell1.conf.txt.gz but without cell1 at | |
| 10:54:43 | vdrok | http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/etc/nova/nova.conf.txt.gz | |
| 10:56:29 | sdague | ok, we'll need dansmith I think. Exactly where the mismatches are here are beyond me. | |
| 10:59:41 | vdrok | same here :( | |
| 11:10:57 | sdague | vdrok: actually, I might have figured it out | |
| 11:11:03 | sdague | we were missing one last conditional | |
| 11:11:11 | sdague | https://review.openstack.org/487809 | |
| 11:12:36 | sdague | I changed the ironic patch to depend on that one | |
| 11:20:00 | takashin | oomichi: Are you around? | |
| 11:23:04 | vdrok | sdague: cool, thank you! | |
| 11:23:29 | sdague | vdrok: fingers crossed | |
| 11:24:09 | Ritesh | Hi, | |
| 11:24:15 | Ritesh | Can anyone please review this patch | |
| 11:24:28 | Ritesh | review pending since long time | |
| 11:24:30 | Ritesh | https://review.openstack.org/#/c/457886/ | |
| 11:32:48 | openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | |
| 11:45:17 | vdrok | sdague: ouch http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cpu.txt.gz#_Jul_27_11_32_29_018076 | |
| 11:47:10 | sdague | vdrok: ok, the conductor didn't get the right config file - http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-pxe_snmp-tinyipa-ubuntu-xenial-nv/e4c3322/logs/screen-n-cond.txt.gz#_Jul_27_11_29_55_156482 | |
| 11:47:20 | sdague | that's solvable | |
| 11:50:53 | sdague | vdrok: ok, lets try again | |
| 11:51:32 | vdrok | yup :) | |
| 11:54:02 | openstackgerrit | Yaroslav proposed openstack/nova master: Added Overbooking DB quota driver https://review.openstack.org/484359 | |
| 12:03:24 | sean-k-mooney | o/ would people object to adding a numa aware varient of reserved_host_memory_mb in queens. we have been seeing OOM errors on our ci systems pereodically for some time even when the system has several GB of free ram. | |
| 12:04:39 | sean-k-mooney | turns out that the OOM killer kicks in when any numa node is out of memory and hte kernel need to alloacte memory there not when the system is out of memory globally | |
| 12:14:18 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 12:21:28 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 12:24:08 | catintheroof | Hi, does anyone has a good guide on how to configure live migration on ocata ? doing the same that worked as of mitaka, doesnt anymore | |
| 12:27:46 | mriedem | oh hi everybody | |
| 12:28:09 | vdrok | sdague: gah, same thing, compute does not get the build instance request http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz | |
| 12:28:15 | vdrok | morning mriedem | |
| 12:29:10 | openstackgerrit | LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128 | |
| 12:29:39 | mriedem | vdrok: did my attempts at dumping the rabbitmqctl report ever work? | |
| 12:30:14 | vdrok | mriedem: seems like that file is not there http://logs.openstack.org/65/487665/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/8cbf580/logs/rabbitmq/ | |
| 12:30:26 | mriedem | yeah...i was getting permission denied errors | |
| 12:31:26 | mriedem | http://logs.openstack.org/64/487664/2/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/c20a804/logs/devstack-gate-cleanup-host.txt | |
| 12:31:35 | mriedem | 2017-07-27 03:38:03.562 | /home/jenkins/workspace/gate-swift-dsvm-functional-ubuntu-xenial-nv/devstack-gate/functions.sh: line 669: /var/log/rabbitmq/cleanup-host-report.txt: Permission denied | |
| 12:31:55 | mriedem | i was thinking maybe because i couldn't write to /var/log/rabbitmq, so i moved it but then it's not even getting called | |
| 12:32:06 | mriedem | https://review.openstack.org/#/c/487664/3/functions.sh | |
| 12:32:23 | mriedem | maybe because we don't have anything in /opt/stack/logs/rabbitmq | |
| 12:33:00 | mriedem | heh, this is wrong anyway | |
| 12:33:01 | mriedem | if [ -f $BASE/logs/rabbitmq/ ]; then | |
| 12:33:04 | mriedem | -f should be -d | |
| 12:34:00 | vdrok | hrm :) | |
| 12:35:39 | sdague | vdrok: well conductor seems like it's working | |
| 12:36:33 | vdrok | is there some kind of diagram for dummies about who calls what in cellsv2 setup? | |
| 12:36:53 | mriedem | vdrok: https://review.openstack.org/#/c/487183 is a start | |
| 12:37:08 | mriedem | vdrok: with singleconductor being set, it should be the same as before the multi-conductor thing | |
| 12:37:15 | vdrok | sdague: yup, that is solved :) | |
| 12:37:20 | vdrok | mriedem: thanks! | |
| 12:37:39 | mriedem | dansmith and i were looking at it last night, and n-cpu, n-sch and n-super-cond are all using the same nova.conf with the same transport_url | |
| 12:38:01 | sdague | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/screen-n-cpu.txt.gz#_Jul_27_12_08_16_707963 - what's up here? | |
| 12:38:03 | mriedem | i traced the build request from super-conductor to scheduler back to super-conductor to the point that super-conductor should rpc cast to nova-compute | |
| 12:38:07 | sdague | That looks like ironic API failed | |
| 12:38:11 | mriedem | but the request never got to n-cpu | |
| 12:38:19 | mriedem | sdague: that's latent | |
| 12:38:36 | sdague | mriedem: oh... would be nice to fix that :P | |
| 12:38:39 | mriedem | sdague: vdrok: opened a bug for that yesterday https://bugs.launchpad.net/ironic/+bug/1706772 | |
| 12:38:39 | openstack | Launchpad bug 1706772 in Ironic "InternalServerError: Internal Server Error (HTTP 500) in n-cpu logs on startup with Ironic driver" [Undecided,Confirmed] | |
| 12:38:49 | mriedem | sdague: sure, but it's some keystone thing in ironic-api | |
| 12:39:05 | mriedem | endpoint discovery blows up | |