| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 18:38:58 | sean-k-mooney | you have enough clearance however that that should not be the issue | |
| 18:39:09 | sean-k-mooney | how many cpu to you have | |
| 18:39:29 | spatel | 32 core | |
| 18:39:53 | nafiux | @mriedem: I was able to launch the first one, but whe I try to launch the second one I got the “Got no allocation candidates from the Placement API.” error | |
| 18:40:39 | sean-k-mooney | you have ~20G free per numa node and you need 12 so it should not be hugepage related | |
| 18:41:15 | sean-k-mooney | you have 32 cores is htat 2 16 core cpus or 2 8 core cpus with hyperthread or 1 32 core cpus | |
| 18:41:25 | mriedem | nafiux: how much disk in the flavor? or are these volume-backed servers? | |
| 18:41:50 | mriedem | nafiux: the placement-api also logs the allocation candidate filtering it does so you should trace the requestid through the placement logs - might need to enable debug | |
| 18:41:58 | spatel | sean-k-mooney: that is what i am trying to find.. what is wrong here :( | |
| 18:42:00 | sean-k-mooney | spatel: it would be useful to know what the vcpu_pin_set is set to in your nova.conf | |
| 18:42:02 | mriedem | that should tell you why it kicked out the provider | |
| 18:42:43 | nafiux | Ouch, I think that is it! Is the hard disk. Which actually doesn’t overcommit. | |
| 18:42:54 | nafiux | I have assigned a lot in the flavor, let me reduce it and test it again. | |
| 18:43:13 | mriedem | yeah your provider only has ~300 GB of disk | |
| 18:43:23 | spatel | grep vcpu_pin_set /etc/nova/nova.conf <--- empty | |
| 18:43:42 | spatel | I have queen cluster and same config works there... | |
| 18:43:48 | spatel | i am running new cluster in stein | |
| 18:43:48 | mriedem | nafiux: if you were doing volume-backed servers then you're probably fine, lots of vcpu and ram on that node | |
| 18:43:50 | mriedem | but very little disk | |
| 18:43:55 | spatel | which causing issue | |
| 18:44:45 | nafiux | @mriedem yes, that is true, only 300 GB free, and the flavor has 160 GB, so only 1 instance with that capacity will be allowed. | |
| 18:45:04 | mriedem | \o/ | |
| 18:46:36 | nafiux | Does "volume-backed" means the actuall physical hard disk of the hypervisor/nova node? | |
| 18:46:48 | mriedem | no | |
| 18:46:55 | mriedem | means the root disk is a cinder volume | |
| 18:47:11 | mriedem | https://docs.openstack.org/nova/latest/user/launch-instance-from-volume.html | |
| 18:48:09 | nafiux | Oh, I don’t have installed cinder. That will help a lot, if I understand correctly? | |
| 18:48:31 | mriedem | i don't know how to answer that question :) | |
| 18:48:46 | mriedem | if you have a compute node with 300 GB of disk and flavors with 160 GB of disk, you're not going to get far | |
| 18:48:54 | mriedem | so you'd have to carve down your flavors to have smaller disk, | |
| 18:49:00 | mriedem | or add more storage to your compute nodes, | |
| 18:49:07 | mriedem | or use cinder to back the root disk | |
| 18:50:02 | nafiux | Lol, yeah. How servers backed with cinder volumes behave? Do they use as needed and not the actual size specified at creation time? | |
| 18:53:13 | mriedem | for volume-backed servers the root gb on the flavor doesn't really apply | |
| 18:53:34 | mriedem | the size of the volume is whatever it is when it's created | |
| 18:53:56 | sean-k-mooney | mriedem: im not sure i did a good job of explaing but it tried to explain the issue again im my reponce to your comments | |
| 18:54:00 | sean-k-mooney | https://review.opendev.org/#/c/669738/2 | |
| 18:54:01 | mriedem | the user can pre-create a volume and provide that to nova when creating the server, or the user can give some parameters to nova and nova will create the volume and attach it to the server | |
| 18:54:16 | sean-k-mooney | but yes i can factor the functional test our into there own regression module | |
| 18:54:27 | mriedem | so many words....but i started that | |
| 18:54:29 | sean-k-mooney | and add teh rebuild/shelve tests | |
| 18:55:31 | mriedem | nafiux: some people use cinder exclusively for root volumes of their servers, some people use shared storage for their compute nodes, some mix both - it takes all kinds | |
| 18:57:56 | sean-k-mooney | nafiux: cinder backed volume behvior depend on teh dirver. some cinder backends support thin provisioning allowing over commit some dont | |
| 18:58:49 | sean-k-mooney | and some time the same dirver change behavior based on your config setting which makes it all the more fun to debug | |
| 18:58:59 | nafiux | Thanks @mridem & @sean-k-mooney. I will read more about those topics, so, in general which are the different options for volumens to the instances? | |
| 19:01:11 | sean-k-mooney | there are two sets of volumes, volumes that have independent lifetime form the vm usually called data volumes | |
| 19:01:20 | sean-k-mooney | thos are create via teh cinder api. | |
| 19:01:40 | sean-k-mooney | and then there are volumes that have ther life time cupled to a vm | |
| 19:02:07 | nafiux | Ok, which is the equivalent to the root volumes in AWS? | |
| 19:02:16 | nafiux | Can I have a root volume with cinder? | |
| 19:02:20 | sean-k-mooney | yes | |
| 19:02:31 | sean-k-mooney | we call that boot form volume or bfv for short | |
| 19:03:01 | sean-k-mooney | in that case you tell nova that the root disk should not be create using the compute nodes default driver and instad should be provide by cinder | |
| 19:03:16 | nafiux | Oh, so, due my current setup for learning, I will leverage cinder for both bfv and additional volumes as needed | |
| 19:03:17 | sean-k-mooney | you can optionl specifythat that volume should be deleted when the vm is or not | |
| 19:04:05 | sean-k-mooney | i wasnt following the full conversation but you can use cinder for both | |
| 19:04:22 | nafiux | Ok, yea, is just because I have a lot of CPU & Memory but few hard disk | |
| 19:04:40 | sean-k-mooney | the default backend for non bfv guest is to create a local file on the compute node for the root disk of the instance | |
| 19:05:12 | sean-k-mooney | you can also tell the compute node to use a differnt default image backend such as ceph | |
| 19:05:46 | sean-k-mooney | in which case nova will create a remote ceph volume out of band of cinder | |
| 19:06:16 | nafiux | Ok, I need to read a lot :D cinder, ceph, and will back later with more specific questions, this is really amazing. | |
| 19:06:18 | sean-k-mooney | other peopl in a similar situation somethime mount the default location where libvirt stores its disk image on nfs | |
| 19:07:32 | sean-k-mooney | so as mriedem said there are lots of way to do it and people do many different things but its better to keep it simple when learning | |
| 19:08:13 | nafiux | Agree. Thanks for the support. | |
| 19:11:28 | mriedem | nafiux: more info on cinder in their docs https://docs.openstack.org/cinder/latest/ | |
| 19:11:46 | mriedem | there is also the ops guide which is older but relevant https://docs.openstack.org/operations-guide/ | |
| 19:12:41 | sean-k-mooney | or if you like video content there are way more videos then you will have time to watch on teh offical youtube channel | |
| 19:12:43 | sean-k-mooney | https://www.youtube.com/user/OpenStackFoundation | |
| 19:13:17 | nafiux | Sure! I will take a look on this too: https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html | |
| 19:47:42 | efried_rollin | mriedem: responded. TL;DR: a Set would be better, but a list will work just fine. | |
| 19:49:33 | efried | I guess that's not FUPable since it's an OVO/RPC thing. No takesy backseys. | |
| 19:49:46 | efried | so if you feel strongly about it you should -1 | |
| 19:52:52 | mriedem | done | |
| 19:53:16 | mriedem | i've made more comments up that series but non-voting b/c i'm a coward | |
| 19:53:27 | mriedem | i'd really like dan to review that series but he's out this week | |
| 19:54:48 | efried | mriedem: FYI Dan was involved earlier, which is how we ended up with the query and filter in their current form. | |
| 19:55:21 | mriedem | yeah i knew he blew up the single megapatch | |
| 19:56:04 | efried | but also was the one who suggested the db function as it now exists. | |
| 19:56:14 | efried | much nicer than it was before | |
| 19:56:43 | mriedem | b/c it's mostly a copy of the one he added for his earlier (first) pre-filters | |
| 19:57:08 | mriedem | fwiw most of my comments are "can you document this?" | |
| 19:57:52 | mriedem | i also feel like forbidden is the wrong word for the user-facing side of this feature, it's really about host isolation | |
| 19:57:57 | mriedem | but it's probably too late to change that now | |
| 19:58:25 | mriedem | unless i'm just too slow to connect the dots, which is possible | |
| 19:58:42 | mriedem | lots of negative logic in this feature makes it hard for me to reason about | |
| 19:58:56 | mriedem | don't not give me the hosts i don't not want | |
| 19:59:04 | efried | I'll have another look at the documentation, but I thought it was ultimately explained the right way for users. | |
| 19:59:17 | efried | The logic is necessarily hairy and compound-negative-confusing | |
| 19:59:25 | efried | but the ux winds up being really crisp and clean imo | |
| 20:02:58 | efried | ...you're right, the doc https://review.opendev.org/#/c/667952/10/doc/source/reference/forbidden-aggregates.rst still talks about forbidden aggregates. Though really, the explanation is very good, and "forbidden aggregates" is really just used as the name-of-feature. | |
| 20:16:30 | mriedem | yeah i'm not -1 on it, it's just a bit confusing | |
| 20:16:33 | mriedem | some of the wording | |
| 20:18:15 | efried | I think you've got a valid point that we could s/forbidden/isolated/ and it would be more comprehensible. The docs could be fixed later, but if we want to rename the conf opt we should do it now. | |
| 20:21:25 | prometheanfire | sean-k-mooney: ya, looks like it's lxml https://review.opendev.org/673848 | |
| 20:23:06 | mriedem | efried: question in https://review.opendev.org/#/c/670182/1 | |
| 20:24:23 | efried | mriedem: responded | |
| 20:29:24 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Fix python3 compatibility of rbd get_fsid https://review.opendev.org/664513 | |
| 20:31:07 | mriedem | amorin: are you still working on https://review.opendev.org/#/c/667294/ ? | |
| 20:45:14 | openstackgerrit | Merged openstack/nova master: neutron: log something more useful in _get_instance_nw_info https://review.opendev.org/673694 | |
| 21:43:21 | efried | mriedem: RequestContext question for ya | |