Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-31
10:52:01 johnthetubaguy lots of API errors
10:52:08 johnthetubaguy bauzas: ack
10:52:10 bauzas and I'm awfully busy
10:52:11 bauzas :(
10:52:22 johnthetubaguy bauzas: understood
10:57:31 johnthetubaguy gmann: I worry about advertising the alias
10:58:39 andymccr johnthetubaguy: yeah i'm trying to understand what a 409 would mean in terms of what is actually conflicting - the endpoint seems ok since it's accepting GET requests (responding with a 200), but any PUT requests are returning bad request 400, and the initial put request returns a 409 conflict
10:59:01 andymccr my understanding is the resource already exists, but im wondering where or what i can do to debug that a bit further
10:59:06 johnthetubaguy andymccr: DB configuration is bad, maybe?
10:59:40 andymccr johnthetubaguy: yeah thats what im thinking - so i noticed the nova_placement db that is created isn't populated, perhaps it needs write permissions that i havn't set up properly
11:00:04 johnthetubaguy andymccr: needs a new nova-manage cmd running
11:00:10 andymccr from a nova perspective i just need to run the nova-manage api_db sync command?
11:01:17 johnthetubaguy andymccr: no
11:01:30 gmann johnthetubaguy: i see your point, and agree that can avoid users using those alias. ll update those in respective attribute description.
11:01:34 johnthetubaguy andymccr: oh... well maybe, I can't remember now actually
11:02:01 andymccr johnthetubaguy: i think i must be missing something because it's not quite happening :)
11:02:09 johnthetubaguy gmann: awesome, thank you
11:02:30 johnthetubaguy andymccr: so do you have it configured the same as the API DB in the conf?
11:02:53 openstackgerrit Bob Ball proposed openstack/nova: Fix live migrate with XenServer https://review.openstack.org/424428
11:03:07 andymccr johnthetubaguy: no there is a separate db
11:03:12 andymccr or rather we've setup a separate database
11:03:18 johnthetubaguy andymccr: OK, that sounds good
11:03:34 andymccr with a separate user
11:03:59 andymccr but then the api_db sync doesn't populate the new db - so i wonder if its a permissions issue, although i imagine it would pull the connection string from nova.conf (for the placement db?)
11:04:34 johnthetubaguy andymccr: oh, interesting. it should pull that from nova.conf, but I wonder if that works
11:07:30 johnthetubaguy andymccr: so that is what devstack does: https://github.com/openstack-dev/devstack/blob/master/lib/placement#L143
11:08:37 openstackgerrit Stephen Finucane proposed openstack/nova: Ignore IOError when creating 'console.log' https://review.openstack.org/392643
11:09:15 andymccr johnthetubaguy: hmm yeah, so that pretty much lines up - i tested with --conf-file but let me test that again
11:09:23 sfinucan artom, mdbooth: ^^^
11:09:46 johnthetubaguy andymccr: this is our nova.conf in the gate: http://logs.openstack.org/63/426163/2/check/gate-tempest-dsvm-neutron-full-ubuntu-trusty/a5ece9b/logs/etc/nova/nova.conf.txt.gz
11:10:39 openstack Launchpad bug 1658877 in OpenStack Compute (nova) "live migration failed with XenServer as hypervisor" [High,In progress] - Assigned to Bob Ball (bob-ball)
11:10:39 BobBall mriedem: See my updates to https://bugs.launchpad.net/nova/+bug/1658877 . I'd very much hope we can still aim for a fix in Ocata as changes to the object are needed and I understand these are particularly difficult to backport?
11:11:09 mdbooth sfinucan: You want to add a test for ignoring EPERM? ;)
11:11:19 johnthetubaguy andymccr: did you configure api_database with the placement database, and use the correct conf file when calling api_sync?
11:11:30 sfinucan mdbooth: Damn - you're quick. Had just started fixing that :)
11:12:05 andymccr johnthetubaguy: yes on the config file, but we have a the placement_database section defined.
11:12:26 johnthetubaguy andymccr: something tells me we screwed up, and nova-manage doesn't read that
11:13:01 andymccr johnthetubaguy: interesting, let me test and remove that section
11:13:06 andymccr i assume it'll just default to the api_database then?
11:13:22 johnthetubaguy andymccr: yeah, we will want to not do that, but it would be good to see that fix your issues
11:14:04 andymccr johnthetubaguy: although - i know the puppet team has dedicated configuration for nova_placement database.
11:14:28 andymccr but lets test
11:14:35 andymccr im somewhat sure its db related
11:14:37 johnthetubaguy andymccr: yeah, we should do a separate database, but it would be good to see if that fixes things
11:16:57 openstackgerrit Takashi NATSUME proposed openstack/nova: Avoid forcing translation on logging calls https://review.openstack.org/413876
11:18:26 johnthetubaguy bauzas: have you noticed how you can't have a separate placement database using the placement_database settings? seems in the gate we test with a single shared API database
11:24:18 openstackgerrit Stephen Finucane proposed openstack/nova: Ignore IOError when creating 'console.log' https://review.openstack.org/392643
11:31:34 openstackgerrit Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163
11:38:54 rpodolyaka johnthetubaguy: yeah, a separate placement db must only be possible with https://review.openstack.org/#/c/362766/
11:40:21 rpodolyaka I wonder if we left placement_database conf group intentionally in this revert - https://review.openstack.org/#/c/360638/ or by mistake
11:42:26 johnthetubaguy rpodolyaka: ah, right, I couldn't see that code when I went digging
11:43:45 johnthetubaguy andymccr: so I think we could make the placement DB separate by having a placement.conf that is used for the nova-manage call and the placement API, where the api-database is configured as the placement database
11:44:04 johnthetubaguy andymccr: seems we are missing some bits on our side for that to work... more sensibly
11:45:10 openstackgerrit Abhishek Kekane proposed openstack/nova: Use is_valid_cidr and is_valid_ipv6_cidr from oslo_utils https://review.openstack.org/320375
11:46:51 andymccr johnthetubaguy: ahh ok - weird. i wonder how the puppet stuff works then - im running a few tests now wrt not using a set placement_db
11:47:09 johnthetubaguy andymccr: +1
11:47:20 andymccr i'll try speak to EmilienM a bit later - he put the pr in for that :)
11:48:18 sfinucan lyarwood: Could you say if this is suitable bugfix to backport to Mitaka? https://review.openstack.org/427119
11:48:31 openstack Launchpad bug 1578155 in OpenStack Compute (nova) newton "duplicate for #1657791 'hw:cpu_thread_policy=prefer' misbehaviour" [Medium,Fix committed] - Assigned to Stephen Finucane (stephenfinucane)
11:48:31 sfinucan Per this bug report https://bugs.launchpad.net/nova/+bug/1657791
11:57:58 tdurakov johnthetubaguy: could you please take a look https://review.openstack.org/#/c/244489/58 especially interested in default rpc method params, see my comments on ps-58, is it really required? I would like to clarify that moment for myself
12:00:58 johnthetubaguy andymccr: on a related note, not sure if you have the cell0 database setup yet, there are details in here: http://docs.openstack.org/developer/nova/cells.html#setup-of-cells-v2
12:01:33 andymccr johnthetubaguy: i was looking at that - is that a requirement? so you have to have a cell0 db (and a cell1 db)
12:01:50 openstackgerrit Stephen Finucane proposed openstack/nova-specs: Add extra-specs-image-meta-conflicts spec https://review.openstack.org/382382
12:02:17 johnthetubaguy andymccr: yes, its all required now, I believe
12:02:23 andymccr johnthetubaguy: i wonder if that is the issue then.
12:02:24 andymccr hmm
12:02:38 andymccr ok cool that gives me some more things to try at the lest
12:02:42 andymccr *least
12:03:22 openstackgerrit Stephen Finucane proposed openstack/nova-specs: Reserve NUMA nodes with PCI devices attached https://review.openstack.org/364468
12:03:34 johnthetubaguy andymccr: you need the API db to know which cell a host is in, else we don't know which cell to send the build request, etc
12:04:25 openstackgerrit Stephen Finucane proposed openstack/nova-specs: Add 'attributed-action-notifications' spec https://review.openstack.org/388573
12:04:58 lyarwood sfinucan: hey sorry was on a call, no I don't think it is http://docs.openstack.org/project-team-guide/stable-branches.html#support-phases
12:06:13 sfinucan lyarwood: I thought as much. I was hoping the clause might apply though :) " It’s nevertheless allowed to backport fixes for other bugs if their safety can be easily proved. "
12:07:44 lyarwood sfinucan: yeah we do make exceptions for simple fixes, I'm just going by the bug subject and Importance posted in the channel
12:10:00 openstack Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]
12:10:00 kashyap Hmm, this has become a bit of blocker it seems -- https://bugs.launchpad.net/bugs/1643911
12:10:57 kashyap And the upstream libvirt folks go: "the stack trace will be totally useless for malloc memory corruption"
12:12:19 openstack Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]
12:12:19 kashyap Because: https://bugs.launchpad.net/nova/+bug/1643911/comments/1
12:12:37 kashyap If you run libvirtd under valgrind in the Gate, then it might hide the problem
12:13:02 kashyap At least libvirt upstream admits: "I mean, we [libvirt] are clearly overwriting the memory somewhere; but stack trace is not helpful in those cases"
12:32:24 openstackgerrit Ghanshyam Mann proposed openstack/nova: Add release note and docs for filter/sort whitelist https://review.openstack.org/421760
12:37:54 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova: Optimize time during PCI alias parsing process https://review.openstack.org/427145
13:07:26 openstackgerrit Emilien Macchi proposed openstack/nova: DNM - tripleo CI test https://review.openstack.org/426949
13:13:19 cdent johnthetubaguy, andymccr: Did you guys work out what was going on a while back with the placement db or are things still weird?
13:13:48 cdent andymccr: I'm pretty sure the original 409 that is happening is because the generation on the resource provider is run, but it corrects itself so the next writes out to work
13:13:53 cdent s/out/ought/
13:14:06 cdent (they do for me when I've been testing things in devstack)
13:14:24 cdent so your theory that it is a database problem is likely
13:17:19 johnthetubaguy cdent: I think the DB wasn't configured correctly
13:17:34 johnthetubaguy cdent: but I could be wrong, not seen andymccr latest results
13:18:02 johnthetubaguy cdent: I forgot we didn't merge the stuff to support the separate placement_database
13:18:59 andymccr johnthetubaguy: cdent - thanks for the info :) I'm testing some more things with database changes/perms etc - will update when i (hopefully) have something working!
13:19:03 cdent yeah, it totally stalled out. I have a working change that I keep up to date for it, but people stated unspecific reservations about it, without specifics there was no way to move on...
13:19:54 johnthetubaguy cdent: jaypipes: I still had worries about the allocations in that ironic patch, I added comments/suggestions
13:20:04 cdent johnthetubaguy: that's next
13:20:26 johnthetubaguy cool, thanks
13:20:28 cdent I skimmed your comments earlier today and my immediate thought was "moar tests"
13:21:12 markus_z mriedem: Would you please remove/re-evaluate your -1 on https://review.openstack.org/#/c/406897/ ?

Earlier   Later