Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-27
01:49:20 mriedem http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/screen-n-cpu.txt.gz#_Jul_26_21_32_50_782647
01:49:20 mriedem 8 seconds before that in n-cpu we're deleting orphan compute nodes
01:49:36 dansmith but, the node doesn't affect the host/topic it goes to
01:50:14 mriedem yeah
01:50:28 mriedem there is a gmr at the end of n-cpu
01:51:27 dansmith transport_url is obscured there too
01:51:35 dansmith also,
01:51:45 dansmith scheduler wouldn't have selected it if it wasn't getting written to the right db,
01:52:28 dansmith but I guess it could be going to the wrong conductor
01:52:55 dansmith so we could try setting the other conductor to the same mq vhost in this scenario, or not start it and see if n-cpu is timing out
01:53:12 dansmith although I don't really see how we could be on the wrong vhost, since we write the config early and don't restart cpu at all
01:56:43 mriedem i was wondering if something was going to the cell1 conductor, but there is nothing in it's logs
01:56:55 mriedem after it dumps the config anyway
01:57:22 dansmith yeah, seems unlikely, but it also has a barebones config, so maybe it's on warning only logs or something?
01:58:29 mriedem it's logging debug messages
01:59:07 dansmith yeah I guess so
01:59:17 dansmith well I dunno, I'm dry on ideas
01:59:33 mriedem there is a rabbit log
01:59:34 mriedem http://logs.openstack.org/58/487458/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/d148ee1/logs/rabbitmq/rabbit@ubuntu-xenial-infracloud-vanilla-10103211.txt.gz
01:59:43 dansmith yeah, but no queue dump right?
01:59:46 mriedem no
01:59:51 mriedem last thing we'd see in super-cond is Jul 26 21:32:58.377797
01:59:58 mriedem and that's where the rabbit log ends
02:01:23 mriedem know how to dump the queue in the rabbitmq logs?
02:02:08 dansmith yeah,
02:02:41 dansmith rabbitmqctl report
02:02:44 dansmith I think gives you a ton of data
02:03:01 dansmith yup, that's all the status dumps in one command
02:03:35 mriedem hmm, but do we just dump that at the very end when collecting logs?
02:03:48 mriedem i know where logs are collected in devstack-gate
02:03:59 dansmith yeah, that will tell us if something is still in a queue
02:04:08 mriedem ok i could tinker with that
02:04:19 mriedem and fix jay's placement 'handle moves' patch
02:05:27 dansmith so the channels section of that dump will have some stuff like consumer_count and messages_unacknowledged
02:05:38 dansmith which would be the things that are sitting from a cast unretrieved
02:05:52 dansmith and then you should see the compute-$hostname topics for anything listening
02:06:06 dansmith not sure that will dump all vhosts
02:06:43 dansmith report doesn't take a vhost, so I imagine it will dump all of them
02:07:02 openstackgerrit Merged openstack/nova master: deprecate ``wsgi_log_format`` config variable https://review.openstack.org/486623
02:07:07 dansmith yeah looks like it
02:08:44 dansmith mriedem: I'm going to retire for the evening but will pick things back up in the morning unless you fix it before then
02:14:20 mriedem adios
02:21:33 melwitt tonyb: excellent memory. I had forgotten about the low hanging fruit etherpad
02:22:09 tonyb melwitt: :D
02:23:20 mriedem https://review.openstack.org/487665
02:48:18 openstackgerrit Matt Riedemann proposed openstack/nova master: placement: account for move operations in claim https://review.openstack.org/487589
02:48:18 openstackgerrit Matt Riedemann proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566
02:48:30 mriedem ^ fixes the random hashseed failures in the unit tests for the bottom change
03:03:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Ensure we unshelve in the cell the instance is mapped https://review.openstack.org/486208
03:05:14 openstackgerrit Merged openstack/nova master: Move the last_bytes util method to libvirt https://review.openstack.org/472228
03:06:10 mikal tonyb: the last_bytes refactor change just passed the gate.
03:08:08 tonyb huzzah!
03:08:22 mikal Gonna upload my smallish change in a minute
03:08:31 mikal And then I'll take a look at that pyhon3.2 thing
03:08:46 tonyb mikal: okay, I'll check it out when it arrives
03:09:27 openstackgerrit Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229
03:09:28 openstackgerrit Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972
03:09:28 openstackgerrit Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166
03:09:29 openstackgerrit Michael Still proposed openstack/nova master: rootwrap.d cleanup mislabeled files https://review.openstack.org/486831
03:09:50 mikal tonyb: my only change there is a relno, so I wouldn't spend a lot of time looking at it
03:16:56 openstackgerrit Merged openstack/nova master: add a retry on DBDeadlock to _set_allocations() https://review.openstack.org/487483
04:22:48 openstackgerrit yuanyue proposed openstack/nova master: Add a periodic task to destroy ReqSpecs of deleted instances https://review.openstack.org/484694
04:55:39 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955
04:58:05 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964
04:58:31 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Add parameters in cold migrate action https://review.openstack.org/410042
04:58:42 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.54 - Enable cold migration with target host https://review.openstack.org/406707
04:59:05 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.54 - List/Show all server migration types https://review.openstack.org/430839
06:11:51 openstackgerrit NidhiMittalHada proposed openstack/python-novaclient master: Help text for "--matching" is not clear. https://review.openstack.org/487708
06:29:05 openstackgerrit jichenjc proposed openstack/nova master: add cli to support matrix https://review.openstack.org/481498
06:30:24 openstackgerrit jichenjc proposed openstack/nova master: Enhance support matrix document https://review.openstack.org/482013
06:31:09 openstackgerrit jichenjc proposed openstack/nova master: Enhance support matrix document https://review.openstack.org/482020
06:31:55 openstackgerrit jichenjc proposed openstack/nova master: Add quiesce and unquiesce in support matrix https://review.openstack.org/482390
06:33:05 openstackgerrit jichenjc proposed openstack/nova master: Add device tag support info in support matrix https://review.openstack.org/481478
06:33:05 openstackgerrit jichenjc proposed openstack/nova master: Add description for tag support in attach https://review.openstack.org/481479
06:40:01 openstackgerrit jichenjc proposed openstack/nova master: WIP:add cert when send keystone request https://review.openstack.org/485121
06:48:08 openstackgerrit jichenjc proposed openstack/nova master: refactory some code on get_password https://review.openstack.org/449458
06:52:45 openstackgerrit LIU Yulong proposed openstack/nova master: Enable reset keypair while rebuilding instance https://review.openstack.org/379128
07:13:56 openstackgerrit Merged openstack/python-novaclient master: Removed extra word 'method' from the NOTE https://review.openstack.org/480988
07:40:17 gibi bauzas: hi! good morning! There is a bugfix that only need a second +2 https://review.openstack.org/#/c/475276/ Could you check it if your time allows?
08:06:38 bauzas gibi: roger, roger.
08:15:54 gibi bauzas: thanks
08:25:58 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send soft_delete from context manager https://review.openstack.org/476459
08:25:59 openstackgerrit Balazs Gibizer proposed openstack/nova master: use context mgr in instance.delete https://review.openstack.org/443764
08:26:01 openstackgerrit Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297
08:43:13 openstackgerrit jichenjc proposed openstack/nova master: refactory some code on get_password https://review.openstack.org/449458
08:44:56 mamandle alex_xu: bauzas: sfinucan: mriedem: please take a look at https://review.openstack.org/#/c/483911/4, comments addressed.
08:51:16 bauzas mamandle: given it's a bugfix and since today is pike-3 milestone day (so FeatureFreeze), I'll prioritize blueprints
08:51:35 bauzas mamandle: that said, just ping me next week and I'll look at it, for sure :)
08:51:56 mamandle bauzas: sure, thanks :)
09:03:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: stabilize test_create_delete_server functional test https://review.openstack.org/487772
09:15:58 vdrok good morning folks!
09:24:16 takashin bauzas: Please review https://review.openstack.org/#/c/408955/. Your comments were addressed.
09:27:35 bauzas takashin: yup, I saw your update
09:28:23 openstackgerrit jichenjc proposed openstack/nova master: add cert when send keystone request https://review.openstack.org/485121
09:28:28 bauzas takashin: I knew you already provided your spec since 2 cycles
09:28:41 bauzas takashin: unfortunately, I didn't have time to review it until last week
09:28:48 bauzas so it's my bad
09:29:34 bauzas takashin: but now we are very close to the FF, would you accept that we could fast-merging that spec for Queens and ask me to be my main priority review for Q ?
09:30:05 bauzas takashin: I really think you worked fine, and adding a new 'forced_destination' seems good with me
09:30:27 takashin bauzas: I want to get it merged by Pike release.

Earlier   Later