| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-21 | |||
| 14:25:19 | mriedem | looking at the CoreFilter code this doesn't make any sense | |
| 14:26:03 | mriedem | we have 8 total, and cpu_allocation_ratio is 16.0, so we end up with vcpus_total = 24, 24-vcpus_used (4) = 20, so we have 20 free vcpus and we're requesting 1 | |
| 14:27:32 | cdent | mriedem: is that a new vm or move/migration/whatever? | |
| 14:27:37 | mriedem | new vm | |
| 14:28:03 | sean-k-mooney | mriedem: well you should be able to use 128 vcps with 8 cpus and an allocation ratios of 16. anyway did you notice this error 'disabled_reason': u'AUTO: Connection to libvirt lost: | |
| 14:28:42 | sean-k-mooney | at http://logs.openstack.org/28/468528/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/337af98/logs/screen-n-sch.txt#_2017-08-21_04_15_14_766 | |
| 14:29:31 | mriedem | hmm seeing that now yeah | |
| 14:30:40 | mriedem | oh duh, | |
| 14:30:48 | mriedem | it's not the CoreFilter, it's the ComputeFilter | |
| 14:30:50 | mriedem | god | |
| 14:31:19 | sean-k-mooney | there are a load of tracebacks in the n-cpu log | |
| 14:31:46 | sean-k-mooney | NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception. | |
| 14:31:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: test allocation handling during scheduler retry https://review.openstack.org/495891 | |
| 14:32:11 | mriedem | sean-k-mooney: that one is a known issue, | |
| 14:32:16 | mriedem | the real problem is libvirt crashed | |
| 14:32:25 | mriedem | so the libvirt driver auto-disables the compute service | |
| 14:35:16 | sean-k-mooney | mriedem: there does not seam to be any usfull info in the libvirtd log other then a bunch of virNetlinkEventCallback spam | |
| 14:35:33 | guimaluf | Could you guys help me to debug this error? http://paste.openstack.org/show/618918/ I'm trying to launch an instance within my testing environment, which consist of 4 vms, keystone/api-controllers, mysql/ceph/rabbitmq, neutron and nova. | |
| 14:36:04 | guimaluf | my concern is mainly focused on HTTPMultipleChoices: HTTPMultipleChoices (HTTP 300) Requested version of OpenStack Images API is not available. I expect nova to contact glance v2 api, but it is trying to reach v1 even with only v2 available | |
| 14:37:46 | mriedem | sean-k-mooney: i won't pursue any further - libvirt crashing is not a new thing in the gate | |
| 14:41:01 | sean-k-mooney | mriedem: well strangly the libvirtd log seams to still be active when the connection drops on the nova side but ya this does not really look like it s novas problem | |
| 14:41:28 | sean-k-mooney | mriedem: the compute agent will reconnect and reactivate itself anyway correct | |
| 14:41:33 | mriedem | yes | |
| 14:42:03 | mriedem | in a prod cloud with hundreds/thousands of nodes, that's probably good enough | |
| 14:42:09 | mriedem | for single node gate jobs it's a killer | |
| 14:43:32 | sean-k-mooney | ya tempest could maybe retry the failed jobs automatically to a certenlimit at the end fo the run in the future but honestly that will likey just slow down the gate when code is actully broken | |
| 14:49:57 | maciejjozefczyk | guimaluf: check nova-scheduler and rabbitmq | |
| 14:50:32 | maciejjozefczyk | guimaluf: s/nova-scheduler/nova-conductor/ | |
| 14:51:17 | mriedem | maciejjozefczyk: anything glance related is likely blowing up in the api | |
| 14:51:21 | mriedem | sounds like misconfiguration | |
| 14:51:32 | mriedem | [glance]api_servers is where i'd look | |
| 14:51:38 | mriedem | in nova.conf | |
| 14:51:56 | mriedem | and make sure that matches what's in the service catalog for the image service | |
| 14:57:34 | maciejjozefczyk | mriedem: Could you check this cherry-pick https://review.openstack.org/#/c/494974/ please :) ? | |
| 14:59:03 | mriedem | maciejjozefczyk: has to go through pike first, | |
| 14:59:19 | mriedem | and i see the cherry pick for pike, but this isn't a regression in pike, so i'm not sure we should include it in the rc2 for pike | |
| 15:01:09 | maciejjozefczyk | mriedem: So it should wait for stable pike? | |
| 15:01:25 | mriedem | maciejjozefczyk: it has to be in stable/pike before we can merge it in stable/ocata, yes | |
| 15:01:32 | mriedem | i'm not sure yet if it should be in rc2 for pike | |
| 15:03:08 | maciejjozefczyk | mriedem: ok | |
| 15:05:47 | mriedem | man i guess you guys were busy last week https://review.openstack.org/#/q/status:merged+project:openstack/nova+branch:stable/pike | |
| 15:06:44 | dansmith | mriedem: anything you see in there that wasn't legit? | |
| 15:06:47 | dtantsur | folks, do you by chance know who is leading https://etherpad.openstack.org/p/queens-PTG-vmbm ? I should have been aware of it, but I'm not :) | |
| 15:07:02 | cdent | dtantsur: probably johnthetubaguy ? | |
| 15:07:03 | dansmith | mriedem: some of the work required some test backports for cleanliness | |
| 15:07:20 | mriedem | dansmith: yeah that looks like the evacuate/shelve offload tests from givi | |
| 15:07:22 | mriedem | *gib | |
| 15:07:23 | mriedem | gdi | |
| 15:07:24 | mriedem | gibi | |
| 15:07:26 | dansmith | yeah | |
| 15:07:42 | dtantsur | johnthetubaguy: is it you? :) | |
| 15:09:12 | mriedem | dansmith: i found one major issue here https://review.openstack.org/#/c/493037/4/nova/compute/resource_tracker.py@1225 | |
| 15:09:39 | dansmith | mriedem: oh the humanity | |
| 15:10:13 | gibi | mriedem: welcome back. :) | |
| 15:13:04 | mriedem | thanks | |
| 15:16:31 | mriedem | it's going to be a rough couple of days so give me softballs if you have them | |
| 15:17:51 | gibi | I'm going for vacation this week so I can promise that I will not file yet another resource allocation bugs. :) | |
| 15:19:01 | mriedem | q | |
| 15:19:04 | mriedem | oops | |
| 15:22:22 | mriedem | ocata 15.0.7 release request https://review.openstack.org/495903 | |
| 15:26:27 | guimaluf | maciejjozefczyk, the error I mentioned is happening on nova-compute node, no errs on conductor logs. I'll enable debug to see if there is something meaningful | |
| 15:28:03 | dansmith | mriedem: I left this for you to make a call on: https://review.openstack.org/#/c/494973/ | |
| 15:29:20 | openstackgerrit | Merged openstack/nova master: replace chance with filter scheduler in func tests https://review.openstack.org/491529 | |
| 15:29:41 | openstackgerrit | Merged openstack/nova master: Clarify that vlan feature means nova-network support https://review.openstack.org/478551 | |
| 15:33:39 | mriedem | dansmith: yeah saw that, wasn't sure since it's not a regression in pike | |
| 15:34:17 | dansmith | mriedem: yeah thought that was the right call, but figured I'd leave it to you.. can wait until after pike releases and then push the backport like normal | |
| 15:45:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove plug_ovs_hybrid, unplug_ovs_hybrid https://review.openstack.org/483030 | |
| 15:48:48 | lvdombrkr | hello guys, im trieng deploy tripleO , and after trying deploy overcloud get error: http://paste.openstack.org/raw/618917/ , in nova scheduler logs i see that tha node is filtred out http://paste.openstack.org/raw/618930/ | |
| 15:48:54 | lvdombrkr | but i dont uderstand why | |
| 15:49:12 | lvdombrkr | any ideas? | |
| 15:50:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Remove useless test https://review.openstack.org/483031 | |
| 15:55:19 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce basic framework for security proxying https://review.openstack.org/345396 | |
| 15:55:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce the VeNCrypt RFB authentication scheme https://review.openstack.org/345398 | |
| 15:55:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce framework for RFB authentication https://review.openstack.org/345397 | |
| 15:55:21 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: provide an RFB security proxy implementation https://review.openstack.org/345399 | |
| 15:57:08 | mriedem | disabling RamFilter by default hits it's first victim https://bugs.launchpad.net/nova/+bug/1712057 | |
| 15:57:09 | openstack | Launchpad bug 1712057 in OpenStack Compute (nova) "When the specified destination host deploys the virtual machine, the allocation ratio is not valid" [Undecided,Invalid] - Assigned to wangyicheng (wang-yicheng) | |
| 15:58:23 | mriedem | dansmith: did anyone mention this https://review.openstack.org/#/c/495811/ ? | |
| 15:58:26 | cfriesen_ | mriedem: whee! do we want to fill in the limits filter somewhere else? | |
| 15:58:38 | cfriesen_ | make that just limits, not filter | |
| 15:58:51 | mriedem | cfriesen_: the intent is to remove all of that stuff eventually, so i don't think so | |
| 15:58:59 | mriedem | the limits are used in the claims code in the compute | |
| 15:59:16 | mriedem | i suspect the reporter of that bug has out of tree code that's relying on the limits | |
| 16:03:16 | cdent | mriedem: what do you mean by “mention this”? gibi posted about it this morning, the author of the new live migration tests is his colleague. gibi’s going on vacation, so “left it to us” | |
| 16:03:43 | mriedem | mention as in, this could be an rc2 blocker | |
| 16:03:47 | mriedem | since it wasn't tagged | |
| 16:03:51 | mriedem | or in the etherpad | |
| 16:05:00 | cdent | I think it’s new since the weekend | |
| 16:09:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove dead function, variable https://review.openstack.org/491512 | |
| 16:15:06 | dansmith | mriedem: mention what? | |
| 16:15:10 | dansmith | mriedem: oh no | |
| 16:17:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Convert 'live_migration_inbound_addr' to HostAddressOpt https://review.openstack.org/456573 | |
| 16:23:18 | lvdombrkr | hello guys, im trieng deploy tripleO , and after trying deploy overcloud get error: http://paste.openstack.org/raw/618917/ , in nova scheduler logs i see that tha node is filtred out http://paste.openstack.org/raw/618930/ | |
| 16:23:29 | lvdombrkr | but i dont uderstand why | |
| 16:23:36 | lvdombrkr | any ideas? | |
| 16:24:55 | stephenfin | lvdombrkr: Looks like you'd be better asking #openstack or #tripleo | |
| 16:28:37 | mriedem | mnestratov|2: fyi https://bugs.launchpad.net/nova/+bug/1711547 | |
| 16:28:38 | openstack | Launchpad bug 1711547 in OpenStack Compute (nova) "Nova service restart disconnects vzstorage volumes" [Undecided,New] | |
| 16:29:16 | mnestratov|2 | thanks | |
| 16:32:40 | mnestratov|2 | we will take care of it | |