| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-07 | |||
| 17:43:06 | bauzas | \o | |
| 17:44:09 | hrw | I am thinking of the same. | |
| 17:48:36 | claudiub|2 | heya. I'm having a problem with the scheduler, if anyone has any ideas. i have 2 compute nodes, but when spawning an instance, the scheduler totally ignores one of them. yes, they're both mapped in a nova cell, and I've tried restarting the nova-compute services / nova-scheduler, and a few hours ago it was fine. | |
| 17:48:41 | claudiub|2 | any ideas? | |
| 17:50:12 | mriedem | claudiub|2: do both have proper inventory in placement? | |
| 17:50:25 | mriedem | openstack resource provider inventory show <compute node uuid> | |
| 17:51:18 | claudiub|2 | checking | |
| 17:51:43 | melwitt | did you do discover_hosts? | |
| 17:52:05 | dansmith | melwitt: presumably being fine hours ago would rule that out | |
| 17:52:17 | claudiub|2 | yep. as I said, they're already mapped to a nova cell, checked through nova-manage cell_v2 list_hosts | |
| 17:52:24 | mriedem | my guess is placement | |
| 17:52:31 | melwitt | oh, sorry I didn't see that part | |
| 17:52:31 | dansmith | mriedem: yep | |
| 17:52:32 | mriedem | they aren't reporting inventory, or it's not what you expect | |
| 17:52:42 | dansmith | or reporting bad inventory somehow | |
| 17:54:38 | claudiub|2 | mriedem: there's no "openstack resource" command apparently | |
| 17:57:26 | mriedem | claudiub|2: you need osc-placement-plugin 1.0.0 | |
| 17:57:45 | mriedem | * https://pypi.python.org/pypi/osc-placement/1.0.0 | |
| 17:58:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround glanceclient bug when CONF.glance.api_servers not set https://review.openstack.org/541008 | |
| 17:59:30 | openstackgerrit | melanie witt proposed openstack/nova master: Make InstanceGroup.get_hosts query all cells https://review.openstack.org/540258 | |
| 18:00:21 | claudiub|2 | yeah, no resource provider | |
| 18:01:11 | mriedem | claudiub|2: openstack resource provider list - doesn't show it? | |
| 18:01:21 | mriedem | check the nova-compute logs then for that host, | |
| 18:01:28 | mriedem | is [placement] set in nova.conf there? | |
| 18:02:50 | claudiub|2 | mriedem: so, openstack resource provider list requires the resource provider's ID, passing in the hypervisor ID gives 404, the other compute node returns a table with memory, cpu, disk. | |
| 18:03:20 | mriedem | ameade: easy bug https://bugs.launchpad.net/placement-osc-plugin/+bug/1747979 | |
| 18:03:21 | openstack | Launchpad bug 1747979 in placement-osc-plugin "docs link in readme is wrong" [Medium,Confirmed] | |
| 18:03:41 | claudiub|2 | as for the nova-compute logs, before I deleted the service from the nova cell and the service-list and restarted, nothing. afterwards, indeed: ResourceProviderCreationFailed: Failed to create resource provider NUCHV12R202 | |
| 18:04:10 | mriedem | "openstack resource provider list" should list resource providers | |
| 18:04:13 | mriedem | and not require an id | |
| 18:04:16 | mriedem | it's a GET /resource_providers call | |
| 18:04:24 | claudiub|2 | mriedem: and i have the placement section in nova.conf | |
| 18:04:25 | mriedem | openstack resource provider show <id> would require an rp uuid | |
| 18:04:34 | mriedem | so figure out why creation of the RP falied | |
| 18:04:36 | mriedem | *failed | |
| 18:04:48 | mriedem | claudiub|2: welcome new contributor, you know you should be in #openstack yeah? :P | |
| 18:05:30 | claudiub|2 | hm, the compute node is in the openstack resource provider list | |
| 18:05:56 | claudiub|2 | well, this is #openstack, it only has a suffix as well. :D | |
| 18:07:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround glanceclient bug when CONF.glance.api_servers not set https://review.openstack.org/541008 | |
| 18:07:16 | mriedem | claudiub|2: then "openstack resource provider inventory show <rp_uuid>" | |
| 18:07:27 | mriedem | does that show vcpu/memory_mb/disk_gb? | |
| 18:10:06 | claudiub|2 | it does. :/ | |
| 18:10:55 | claudiub|2 | anyways gonna check why the exception occurs for now | |
| 18:11:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474 | |
| 18:11:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160 | |
| 18:11:53 | mriedem | claudiub|2: you can also run, i think, "openstack resource provider show --allocations <rp_uuid>" to see if placement thinks there are instances on the node still | |
| 18:12:05 | mriedem | might need to check the CLI help for the proper command option | |
| 18:12:25 | dansmith | mriedem: these compute rpc bumps are passing now and ready for you to sh*t on them: https://review.openstack.org/#/c/541005 | |
| 18:12:44 | mriedem | dansmith: let me go eat some vietnamese quick and work up a healthy dump | |
| 18:13:13 | melwitt | o.O | |
| 18:13:14 | dansmith | um, thanks? | |
| 18:13:22 | mriedem | you wanted me to shit on them | |
| 18:13:24 | mriedem | i need ammo | |
| 18:13:31 | dansmith | no, I don't want you to, I just know you will | |
| 18:13:50 | mriedem | do'nt act like you're not into it | |
| 18:14:00 | claudiub|2 | hm, a lot of allocations it seems. | |
| 18:14:18 | mriedem | claudiub|2: ok, so something blew up on instance delete (or something) and allocations aren't getting cleaned up | |
| 18:14:20 | claudiub|2 | even though i only have 1 instance | |
| 18:40:38 | stvnoyes | mriedem: create multiple servers with a bdm bug is filed - https://bugs.launchpad.net/nova/+bug/1747985 | |
| 18:40:39 | openstack | Launchpad bug 1747985 in OpenStack Compute (nova) "Create Multiple Servers fails when a volume is specified" [Undecided,New] | |
| 18:41:53 | stvnoyes | mriedem: I going on to see what else I hit. it doesn't just work once that check is removed. fyi | |
| 19:29:51 | hrw | can someone review https://review.openstack.org/#/c/541728/ and decide is it worth second +2? | |
| 19:51:21 | cdent | mriedem: did you see this bit of my FML: https://review.openstack.org/#/c/541832/ | |
| 20:01:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: XenAPI: Provide support matrix and doc for VGPU https://review.openstack.org/540808 | |
| 20:03:38 | mriedem | cdent: nope | |
| 20:03:41 | mriedem | just got back from lunch | |
| 20:03:48 | cdent | such shame | |
| 20:04:53 | mriedem | oh i see it's your fault | |
| 20:04:54 | mriedem | geez | |
| 20:05:27 | cdent | mriedem: but once I fixed that I started running with 20 hypervisors and launching 1000s of instances. ran into the rpc timeouts you did. rabbit is a pig, the conductors eats all the cpu, placement does not sweat | |
| 20:05:32 | cdent | yeah, my bad, three years ago | |
| 20:05:51 | mriedem | cdent: rpc timeouts for select_destinations? | |
| 20:06:05 | cdent | hold, lemme check | |
| 20:06:37 | mriedem | that's what i was hitting at sone point, b/c select_destinations is an rpc call, and when you're booting 1000 instances at once, we timeout on that call | |
| 20:06:40 | dansmith | cdent: I think devstack by default configures few workers per core which is unlike a real deployment, so with thousands of real builds you'll be pushing pretty hard on a small system | |
| 20:06:55 | mriedem | devstack is i think ncpu / 2 | |
| 20:06:56 | mriedem | for each service | |
| 20:07:00 | mriedem | that has multiple workers | |
| 20:07:16 | dansmith | yeah | |
| 20:07:22 | mriedem | if you did ncpu workers per service, which devstack used to do, you get OOM | |
| 20:07:25 | dansmith | right | |
| 20:08:20 | cdent | yes in select_destinations | |
| 20:08:48 | mriedem | yeah for that issue i had put some thoughts in https://review.openstack.org/#/c/510235/ | |
| 20:08:56 | mriedem | but very much quarter baked | |
| 20:09:48 | cdent | dansmith: I'm multinode, using the fake driver, and I'm in the process of now of moving all the nova-compute action to a separate host. the control plane host is 16GB and 16 cores | |
| 20:10:07 | cdent | rabbit is eating a _ton_ of ram, followed by mysql | |
| 20:10:08 | dansmith | ah fake driver, well, that might be different | |
| 20:10:28 | dansmith | cdent: do you have notifications disabled? | |
| 20:10:38 | cdent | no, but that's an excellent idea | |
| 20:10:39 | dansmith | cdent: otherwise I think you're just filling a queue with no consumer | |
| 20:10:50 | cdent | thanks | |
| 20:10:58 | dansmith | thank me if/when it matters :P | |
| 20:11:29 | mriedem | that's something i've been wondering about benchmarking lately, | |
| 20:11:34 | mriedem | with and without versioned notifications enabled | |
| 20:11:44 | mriedem | by default, we have both versioned and unversioned notifications enabled, | |
| 20:11:56 | mriedem | and versioned notifications do a lot of lazy-loading of fields to build the notification payload | |
| 20:11:58 | cdent | well it's a good idea nonetheless as I'm trying to remove as much stuff I don't care about™ as possible | |
| 20:12:06 | cdent | and at the moment, notifications are basically noise | |
| 20:12:20 | mriedem | yeah - take a note of what impact disabling that makes if you can | |
| 20:12:39 | mriedem | i.e. we might want to consider changing the default | |