| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 00:18:19 | mikal | That didn't | |
| 00:18:24 | mikal | But it seems to hate merging | |
| 00:19:03 | tonyb | the py35 sdvm job seems to fail lots but perhaps it just hates us | |
| 00:19:06 | mikal | Hmmm, 472228 is in the check queue not the gate queue? | |
| 00:19:20 | tonyb | mikal: we had to recheck | |
| 00:19:21 | mikal | Looks like its going to pass there though | |
| 00:19:29 | tonyb | which does both queues | |
| 00:19:33 | mikal | Does it go to the gate queue automagically? | |
| 00:19:39 | mikal | Oh, so it runs all the tests twice? | |
| 00:19:46 | mikal | Thus maximizing the changes of a flakey test? | |
| 00:19:55 | tonyb | if it still has +W it will just move to gate when it gets a +1 from jenkins | |
| 00:20:35 | tonyb | mikal: Yeah but if anything has merged since the check was run then the results could now be wrong so it has to do bith | |
| 00:20:40 | tonyb | *both* | |
| 00:21:10 | mikal | So why not just re-run only the gate job? | |
| 00:22:48 | tonyb | 'cause gate and check have different configs and you could effectivle bypass a job that will fail but not catch it | |
| 00:24:34 | mikal | I hate everything | |
| 00:47:59 | openstackgerrit | yuanyue proposed openstack/nova master: Add a periodic task to destroy ReqSpecs of deleted instances https://review.openstack.org/484694 | |
| 01:08:15 | mikal | tonyb: check passed | |
| 01:08:29 | rm_work | anyone know what the var for the nova-cpu config file is in devstack | |
| 01:08:30 | rm_work | like | |
| 01:08:39 | rm_work | https://github.com/problemv/devstack_deploy/blob/master/local.conf#L31-L34 | |
| 01:08:49 | rm_work | I have that but it needs to be in nova-cpu conf too i think? | |
| 01:08:55 | rm_work | config file split recently? | |
| 01:10:03 | tonyb | rm_work: That should work. What are you seeing as a problme | |
| 01:10:07 | tonyb | mikal: \o/ | |
| 01:10:13 | rm_work | tonyb: it doesn't ... use it? | |
| 01:10:18 | rm_work | apparently there is another conf now | |
| 01:10:54 | rm_work | https://www.irccloud.com/pastebin/8fg3XGaC/ | |
| 01:11:25 | rm_work | adding it to nova-cpu.conf seems to work | |
| 01:11:30 | rm_work | but i don't know how to automate it | |
| 01:11:34 | rm_work | $NOVA_CPU_CONF maybe? | |
| 01:13:21 | tonyb | rm_work: where are you getting devstack from? | |
| 01:15:22 | tonyb | rm_work: http://paste.openstack.org/show/616640 | |
| 01:15:39 | tonyb | rm_work: I don't see anything in devstack that is splitting things like that | |
| 01:16:18 | rm_work | so you don't have a nova-cpu.conf? | |
| 01:17:22 | tonyb | rm_work: nope: http://logs.openstack.org/28/472228/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/91e93eb/logs/etc/nova/ | |
| 01:17:38 | rm_work | http://logs.openstack.org/28/472228/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/91e93eb/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 01:17:42 | tonyb | rm_work: Oh wait it is there | |
| 01:17:47 | rm_work | with the libvirt section | |
| 01:18:02 | rm_work | as of ... recently maybe? | |
| 01:21:40 | tonyb | rm_work: You're right NOVA_CPU_CONF http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n54 | |
| 01:21:48 | rm_work | cool :) good guess | |
| 01:22:36 | mriedem | rm_work: tonyb: merged yesterday | |
| 01:22:50 | mriedem | check the dev list about multi-tier conductor | |
| 01:22:55 | tonyb | mriedem: Oh I'm not that behind then | |
| 01:23:28 | rm_work | yeah, octavia folks are just *bleeding edge* :P | |
| 01:23:34 | rm_work | we always run into this stuff first | |
| 01:23:35 | tonyb | rm_work: my first grep didn't checkout master (it only downlaoded it) :( | |
| 01:24:51 | tonyb | mriedem: oh the conductor fleet stuff | |
| 01:25:45 | mriedem | yeah | |
| 01:25:59 | mriedem | a change merged today that allows you to disable that behavior if you need to | |
| 01:26:15 | mriedem | https://review.openstack.org/#/c/487485/ | |
| 01:29:08 | mriedem | although, even with that set to singleconductor, i'm seeing the superconductor running http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/ | |
| 01:29:16 | mriedem | dansmith: ^ maybe why the ironic job is still failing? | |
| 01:29:41 | dansmith | mriedem: in single conductor, the super conductor is the only one that gets hit | |
| 01:30:07 | mriedem | but we still start the cell conductor? | |
| 01:30:19 | mriedem | i'm seeing a request go through the scheduler, it picks a host, but then i don't see the request going through n-cpu | |
| 01:30:19 | dansmith | yeah, it was a hack for grenade remember | |
| 01:32:09 | mriedem | req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 | |
| 01:32:17 | mriedem | http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/screen-n-super-cond.txt.gz#_Jul_26_21_32_58_412591 | |
| 01:32:22 | mriedem | super conductor just seems to die there | |
| 01:32:52 | mriedem | and i never see req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 show up in n-cpu logs | |
| 01:33:22 | dansmith | is that req a boot request? | |
| 01:33:33 | dansmith | you'd never see it hit cpu if it got novalidhost right? | |
| 01:33:52 | mriedem | it didn't get novalidhost | |
| 01:33:57 | mriedem | i see n-sch pick a host | |
| 01:34:07 | mriedem | and yeah that's the boot request | |
| 01:34:15 | mriedem | super-cond, sch and cpu are all using nova.conf | |
| 01:34:24 | mriedem | which points at transport_url = rabbit://stackrabbit:secretrabbit@15.184.66.253:5672/ | |
| 01:34:52 | mriedem | db connection is set to nova_cell1 | |
| 01:34:53 | dansmith | yeah, which is a pretty normal setup | |
| 01:35:09 | dansmith | is it normal to have cpu deleting tons of orphan nodes? | |
| 01:36:08 | dansmith | and this is single node so no chance we're missing logs from another conductor I guess | |
| 01:36:59 | mriedem | lots of ComputeHostNotFound in the n-cpu logs | |
| 01:37:41 | mriedem | this is the host/node that's picked in the scheduler | |
| 01:37:41 | mriedem | Selected host: (ubuntu-xenial-infracloud-vanilla-10103211, eda7dc99-9e17-479c-b234-9d87956e9c56) ram: 384MB disk: 10240MB io_ops: 0 instances: 0 | |
| 01:37:42 | dansmith | so you think conductor got the selected host and cast to compute but it disappeared? | |
| 01:37:53 | mriedem | or conductor didn't get it.. | |
| 01:38:00 | dansmith | unfortunately, we don't get to see the _actual_ trnsport_url loaded in compute | |
| 01:38:07 | dansmith | that req showed up in conductor, no? | |
| 01:38:35 | dansmith | yeah, it clearly got to the conductor | |
| 01:39:07 | mriedem | yeah i see block_device_mapping in conductor | |
| 01:39:12 | mriedem | which comes after we get the scheduler response | |
| 01:39:37 | mriedem | http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/screen-n-super-cond.txt.gz#_Jul_26_21_32_58_377797 | |
| 01:40:20 | mriedem | i never see "Starting instance" in the n-cpu logs | |
| 01:40:24 | dansmith | well, they're all using the same config, and the config is good, I don't really see how it's breaking between conductor and compute | |
| 01:40:29 | mriedem | so it looks like the request never gets to build_and_run_instance in n-cpu | |
| 01:40:35 | mriedem | me neither | |
| 01:40:39 | dansmith | yeah | |
| 01:40:40 | dansmith | but I dunno how to explain that | |
| 01:43:08 | dansmith | the hostname and node id seem to match up between compute and what scheduler selected | |
| 01:43:34 | dansmith | if it picked a host that didn't exist we would cast to the wrong topic and it'd get dropped, but.. | |
| 01:43:38 | dansmith | they match | |
| 01:45:39 | dansmith | no rabbit dump at the end so we can't see if there are different topics or messages sitting in queues | |
| 01:48:02 | mriedem | ok so scheduler picks the host here | |
| 01:48:02 | mriedem | Jul 26 21:32:57.959669 ubuntu-xenial-infracloud-vanilla-10103211 nova-scheduler[12180]: DEBUG nova.scheduler.filter_scheduler [None req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 tempest-BaremetalBasicOps-1511666062 tempest-BaremetalBasicOps-1511666062] Selected host: (ubuntu-xenial-infracloud-vanilla-10103211, eda7dc99-9e17-479c-b234-9d87956e9c56) ram: 384MB disk: 10240MB io_ops: 0 instances: 0 {{(pid=12180) _schedule /opt/stack/n | |
| 01:48:02 | mriedem | ova/nova/scheduler/filter_scheduler.py:157}} | |
| 01:48:25 | mriedem | super-conductor creates the bdms in cell1 here | |
| 01:48:26 | mriedem | Jul 26 21:32:58.377797 ubuntu-xenial-infracloud-vanilla-10103211 nova-conductor[13817]: DEBUG nova.conductor.manager [None req-b1d7d389-f01a-4ea6-8de4-621fc28f7fc7 tempest-BaremetalBasicOps-1511666062 tempest-BaremetalBasicOps-1511666062] [instance: 6e534573-5533-41d6-b9d7-28bfd7dbfe2d] block_device_mapping [BlockDeviceMapping(attachment_id=<?>,boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,delete | |
| 01:48:27 | mriedem | >,deleted_at=<?>,destination_type='local',device_name=None,device_type='disk',disk_bus=None,guest_format=None,id=<?>,image_id='2120252b-74dd-46b0-8649-ef7bcbdd0f50',instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=<?>,volume_id=None,volume_size=None)] {{(pid=14569) _create_block_device_mapping /opt/stack/new/nova/nova/conductor/manager.py:829}} | |