| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-31 | |||
| 10:52:11 | bauzas | :( | |
| 10:52:22 | johnthetubaguy | bauzas: understood | |
| 10:57:31 | johnthetubaguy | gmann: I worry about advertising the alias | |
| 10:58:39 | andymccr | johnthetubaguy: yeah i'm trying to understand what a 409 would mean in terms of what is actually conflicting - the endpoint seems ok since it's accepting GET requests (responding with a 200), but any PUT requests are returning bad request 400, and the initial put request returns a 409 conflict | |
| 10:59:01 | andymccr | my understanding is the resource already exists, but im wondering where or what i can do to debug that a bit further | |
| 10:59:06 | johnthetubaguy | andymccr: DB configuration is bad, maybe? | |
| 10:59:40 | andymccr | johnthetubaguy: yeah thats what im thinking - so i noticed the nova_placement db that is created isn't populated, perhaps it needs write permissions that i havn't set up properly | |
| 11:00:04 | johnthetubaguy | andymccr: needs a new nova-manage cmd running | |
| 11:00:10 | andymccr | from a nova perspective i just need to run the nova-manage api_db sync command? | |
| 11:01:17 | johnthetubaguy | andymccr: no | |
| 11:01:30 | gmann | johnthetubaguy: i see your point, and agree that can avoid users using those alias. ll update those in respective attribute description. | |
| 11:01:34 | johnthetubaguy | andymccr: oh... well maybe, I can't remember now actually | |
| 11:02:01 | andymccr | johnthetubaguy: i think i must be missing something because it's not quite happening :) | |
| 11:02:09 | johnthetubaguy | gmann: awesome, thank you | |
| 11:02:30 | johnthetubaguy | andymccr: so do you have it configured the same as the API DB in the conf? | |
| 11:02:53 | openstackgerrit | Bob Ball proposed openstack/nova: Fix live migrate with XenServer https://review.openstack.org/424428 | |
| 11:03:07 | andymccr | johnthetubaguy: no there is a separate db | |
| 11:03:12 | andymccr | or rather we've setup a separate database | |
| 11:03:18 | johnthetubaguy | andymccr: OK, that sounds good | |
| 11:03:34 | andymccr | with a separate user | |
| 11:03:59 | andymccr | but then the api_db sync doesn't populate the new db - so i wonder if its a permissions issue, although i imagine it would pull the connection string from nova.conf (for the placement db?) | |
| 11:04:34 | johnthetubaguy | andymccr: oh, interesting. it should pull that from nova.conf, but I wonder if that works | |
| 11:07:30 | johnthetubaguy | andymccr: so that is what devstack does: https://github.com/openstack-dev/devstack/blob/master/lib/placement#L143 | |
| 11:08:37 | openstackgerrit | Stephen Finucane proposed openstack/nova: Ignore IOError when creating 'console.log' https://review.openstack.org/392643 | |
| 11:09:15 | andymccr | johnthetubaguy: hmm yeah, so that pretty much lines up - i tested with --conf-file but let me test that again | |
| 11:09:23 | sfinucan | artom, mdbooth: ^^^ | |
| 11:09:46 | johnthetubaguy | andymccr: this is our nova.conf in the gate: http://logs.openstack.org/63/426163/2/check/gate-tempest-dsvm-neutron-full-ubuntu-trusty/a5ece9b/logs/etc/nova/nova.conf.txt.gz | |
| 11:10:39 | BobBall | mriedem: See my updates to https://bugs.launchpad.net/nova/+bug/1658877 . I'd very much hope we can still aim for a fix in Ocata as changes to the object are needed and I understand these are particularly difficult to backport? | |
| 11:10:39 | openstack | Launchpad bug 1658877 in OpenStack Compute (nova) "live migration failed with XenServer as hypervisor" [High,In progress] - Assigned to Bob Ball (bob-ball) | |
| 11:11:09 | mdbooth | sfinucan: You want to add a test for ignoring EPERM? ;) | |
| 11:11:19 | johnthetubaguy | andymccr: did you configure api_database with the placement database, and use the correct conf file when calling api_sync? | |
| 11:11:30 | sfinucan | mdbooth: Damn - you're quick. Had just started fixing that :) | |
| 11:12:05 | andymccr | johnthetubaguy: yes on the config file, but we have a the placement_database section defined. | |
| 11:12:26 | johnthetubaguy | andymccr: something tells me we screwed up, and nova-manage doesn't read that | |
| 11:13:01 | andymccr | johnthetubaguy: interesting, let me test and remove that section | |
| 11:13:06 | andymccr | i assume it'll just default to the api_database then? | |
| 11:13:22 | johnthetubaguy | andymccr: yeah, we will want to not do that, but it would be good to see that fix your issues | |
| 11:14:04 | andymccr | johnthetubaguy: although - i know the puppet team has dedicated configuration for nova_placement database. | |
| 11:14:28 | andymccr | but lets test | |
| 11:14:35 | andymccr | im somewhat sure its db related | |
| 11:14:37 | johnthetubaguy | andymccr: yeah, we should do a separate database, but it would be good to see if that fixes things | |
| 11:16:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova: Avoid forcing translation on logging calls https://review.openstack.org/413876 | |
| 11:18:26 | johnthetubaguy | bauzas: have you noticed how you can't have a separate placement database using the placement_database settings? seems in the gate we test with a single shared API database | |
| 11:24:18 | openstackgerrit | Stephen Finucane proposed openstack/nova: Ignore IOError when creating 'console.log' https://review.openstack.org/392643 | |
| 11:31:34 | openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | |
| 11:38:54 | rpodolyaka | johnthetubaguy: yeah, a separate placement db must only be possible with https://review.openstack.org/#/c/362766/ | |
| 11:40:21 | rpodolyaka | I wonder if we left placement_database conf group intentionally in this revert - https://review.openstack.org/#/c/360638/ or by mistake | |
| 11:42:26 | johnthetubaguy | rpodolyaka: ah, right, I couldn't see that code when I went digging | |
| 11:43:45 | johnthetubaguy | andymccr: so I think we could make the placement DB separate by having a placement.conf that is used for the nova-manage call and the placement API, where the api-database is configured as the placement database | |
| 11:44:04 | johnthetubaguy | andymccr: seems we are missing some bits on our side for that to work... more sensibly | |
| 11:45:10 | openstackgerrit | Abhishek Kekane proposed openstack/nova: Use is_valid_cidr and is_valid_ipv6_cidr from oslo_utils https://review.openstack.org/320375 | |
| 11:46:51 | andymccr | johnthetubaguy: ahh ok - weird. i wonder how the puppet stuff works then - im running a few tests now wrt not using a set placement_db | |
| 11:47:09 | johnthetubaguy | andymccr: +1 | |
| 11:47:20 | andymccr | i'll try speak to EmilienM a bit later - he put the pr in for that :) | |
| 11:48:18 | sfinucan | lyarwood: Could you say if this is suitable bugfix to backport to Mitaka? https://review.openstack.org/427119 | |
| 11:48:31 | sfinucan | Per this bug report https://bugs.launchpad.net/nova/+bug/1657791 | |
| 11:48:31 | openstack | Launchpad bug 1578155 in OpenStack Compute (nova) newton "duplicate for #1657791 'hw:cpu_thread_policy=prefer' misbehaviour" [Medium,Fix committed] - Assigned to Stephen Finucane (stephenfinucane) | |
| 11:57:58 | tdurakov | johnthetubaguy: could you please take a look https://review.openstack.org/#/c/244489/58 especially interested in default rpc method params, see my comments on ps-58, is it really required? I would like to clarify that moment for myself | |
| 12:00:58 | johnthetubaguy | andymccr: on a related note, not sure if you have the cell0 database setup yet, there are details in here: http://docs.openstack.org/developer/nova/cells.html#setup-of-cells-v2 | |
| 12:01:33 | andymccr | johnthetubaguy: i was looking at that - is that a requirement? so you have to have a cell0 db (and a cell1 db) | |
| 12:01:50 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs: Add extra-specs-image-meta-conflicts spec https://review.openstack.org/382382 | |
| 12:02:17 | johnthetubaguy | andymccr: yes, its all required now, I believe | |
| 12:02:23 | andymccr | johnthetubaguy: i wonder if that is the issue then. | |
| 12:02:24 | andymccr | hmm | |
| 12:02:38 | andymccr | ok cool that gives me some more things to try at the lest | |
| 12:02:42 | andymccr | *least | |
| 12:03:22 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs: Reserve NUMA nodes with PCI devices attached https://review.openstack.org/364468 | |
| 12:03:34 | johnthetubaguy | andymccr: you need the API db to know which cell a host is in, else we don't know which cell to send the build request, etc | |
| 12:04:25 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs: Add 'attributed-action-notifications' spec https://review.openstack.org/388573 | |
| 12:04:58 | lyarwood | sfinucan: hey sorry was on a call, no I don't think it is http://docs.openstack.org/project-team-guide/stable-branches.html#support-phases | |
| 12:06:13 | sfinucan | lyarwood: I thought as much. I was hoping the clause might apply though :) " It’s nevertheless allowed to backport fixes for other bugs if their safety can be easily proved. " | |
| 12:07:44 | lyarwood | sfinucan: yeah we do make exceptions for simple fixes, I'm just going by the bug subject and Importance posted in the channel | |
| 12:10:00 | kashyap | Hmm, this has become a bit of blocker it seems -- https://bugs.launchpad.net/bugs/1643911 | |
| 12:10:00 | openstack | Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] | |
| 12:10:57 | kashyap | And the upstream libvirt folks go: "the stack trace will be totally useless for malloc memory corruption" | |
| 12:12:19 | kashyap | Because: https://bugs.launchpad.net/nova/+bug/1643911/comments/1 | |
| 12:12:19 | openstack | Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] | |
| 12:12:37 | kashyap | If you run libvirtd under valgrind in the Gate, then it might hide the problem | |
| 12:13:02 | kashyap | At least libvirt upstream admits: "I mean, we [libvirt] are clearly overwriting the memory somewhere; but stack trace is not helpful in those cases" | |
| 12:32:24 | openstackgerrit | Ghanshyam Mann proposed openstack/nova: Add release note and docs for filter/sort whitelist https://review.openstack.org/421760 | |
| 12:37:54 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova: Optimize time during PCI alias parsing process https://review.openstack.org/427145 | |
| 13:07:26 | openstackgerrit | Emilien Macchi proposed openstack/nova: DNM - tripleo CI test https://review.openstack.org/426949 | |
| 13:13:19 | cdent | johnthetubaguy, andymccr: Did you guys work out what was going on a while back with the placement db or are things still weird? | |
| 13:13:48 | cdent | andymccr: I'm pretty sure the original 409 that is happening is because the generation on the resource provider is run, but it corrects itself so the next writes out to work | |
| 13:13:53 | cdent | s/out/ought/ | |
| 13:14:06 | cdent | (they do for me when I've been testing things in devstack) | |
| 13:14:24 | cdent | so your theory that it is a database problem is likely | |
| 13:17:19 | johnthetubaguy | cdent: I think the DB wasn't configured correctly | |
| 13:17:34 | johnthetubaguy | cdent: but I could be wrong, not seen andymccr latest results | |
| 13:18:02 | johnthetubaguy | cdent: I forgot we didn't merge the stuff to support the separate placement_database | |
| 13:18:59 | andymccr | johnthetubaguy: cdent - thanks for the info :) I'm testing some more things with database changes/perms etc - will update when i (hopefully) have something working! | |
| 13:19:03 | cdent | yeah, it totally stalled out. I have a working change that I keep up to date for it, but people stated unspecific reservations about it, without specifics there was no way to move on... | |
| 13:19:54 | johnthetubaguy | cdent: jaypipes: I still had worries about the allocations in that ironic patch, I added comments/suggestions | |
| 13:20:04 | cdent | johnthetubaguy: that's next | |
| 13:20:26 | johnthetubaguy | cool, thanks | |
| 13:20:28 | cdent | I skimmed your comments earlier today and my immediate thought was "moar tests" | |
| 13:21:12 | markus_z | mriedem: Would you please remove/re-evaluate your -1 on https://review.openstack.org/#/c/406897/ ? | |
| 13:22:40 | openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | |
| 13:23:06 | openstackgerrit | Sylvain Bauza proposed openstack/nova: Add a status check for enabled filters https://review.openstack.org/427200 | |
| 13:24:47 | sfinucan | Kevin_Zheng: Could you take a look at https://bugs.launchpad.net/nova/+bug/1660317 ? fwict, the error message that they're receiving should be mitigated by a patch of yours | |