| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-09 | |||
| 13:10:45 | shilpasd | Hi need bit input, i am currently working on implementation of https://review.opendev.org/#/c/650188/, Allow compute nodes to use shared storage provider for DISK_GB resources | |
| 13:11:24 | shilpasd | Gone through RESHAPE functionality implemented and here which deals with reshaping inventory and allocations | |
| 13:11:33 | shilpasd | In Shared storage Provider, similarly doing reshape where moving allocations of DISK_GB from compute node to shared storage provider | |
| 13:12:10 | shilpasd | now i want to correct local_gb_used at compute node at nova side | |
| 13:12:23 | shilpasd | so can you help me how to proceed here | |
| 13:13:34 | shilpasd | pre_start_hook() first update_available_resource() and then calls update_provider_tree() | |
| 13:14:13 | shilpasd | and during update_provider_tree() i am doing RESHAPE for DISK_GB similarly done for VGPU | |
| 13:16:01 | efried | shilpasd: let me go look at some code... | |
| 13:16:15 | efried | IIRC local_gb_used is a bit tricky | |
| 13:17:59 | efried | shilpasd: I'm sort of thinking we may want local_gb_used to be set to zero when the host is using shared disk. | |
| 13:18:10 | efried | mriedem, gibi, dansmith: You guys have a take on this? ^ | |
| 13:24:28 | mriedem | idk, i'd think local_gb_used would at least report reserved_host_disk_mb (which defaults to 0) | |
| 13:25:19 | mriedem | otherwise i'd tend to agree that it should be 0 for instance usage in the resource tracker if you're using shared storage on that node | |
| 13:25:39 | mriedem | mdbooth might have other wrinkles in mind there | |
| 13:28:42 | dansmith | yea, what mriedem said... because in the future we could have the compute *also* reporting its own disk inventory if it gained the ability to do that thing | |
| 13:28:48 | mdbooth | Hmm, I'd need to look at code. Depends how local_gb_used is used. | |
| 13:36:43 | efried | mdbooth: The only nod to thin provisioning is allocation_ratio | |
| 13:36:50 | efried | which amounts to a wild-ass guess | |
| 13:37:19 | efried | and is why cinder has shown minimal enthusiasm for adopting placement to do their tracking | |
| 13:37:22 | mdbooth | IIRC local_gb_used is all about thin provisioning. | |
| 13:37:37 | efried | oh? | |
| 13:37:52 | mdbooth | IIRC it answers the question: can I migrate here? | |
| 13:38:56 | mdbooth | It's weird that we don't track actual usage in placement. | |
| 13:39:40 | efried | mdbooth: you would have to be sending updates all the time | |
| 13:40:00 | mdbooth | Yeah | |
| 13:40:04 | efried | the actual usage of thin-provisioned storage changes every time the guest does a write | |
| 13:40:23 | efried | also, somebody told me that the storage doesn't ever actually tell you how much space it really has anyway. | |
| 13:40:31 | efried | the more expensive the storage, the less accurate | |
| 13:40:32 | mdbooth | *some* storage | |
| 13:40:44 | efried | which seems bizarre to me | |
| 13:40:46 | mdbooth | We don't use that in Nova, though. | |
| 13:40:52 | efried | "we" don't? | |
| 13:41:10 | efried | I was assuming we would eventually be using this shared storage stuff for cinder volumes too | |
| 13:41:10 | mdbooth | The question we want to answer is: "do I have enough space to write X bytes here"? | |
| 13:41:25 | mdbooth | It seems weird to me that any useful storage would not be able to answer that question. | |
| 13:41:32 | efried | agree with that | |
| 13:41:35 | efried | but apparently it is so | |
| 13:41:35 | dansmith | mdbooth: uh, we do if we're on nfs backed by it | |
| 13:41:48 | mdbooth | That's a different question to "how much storage do I have?" | |
| 13:41:51 | dansmith | but the most expensive and most impressive storage gives you no useful numbers | |
| 13:42:15 | dansmith | reportedly some of them give you nothing more than "buy more disks around october" | |
| 13:42:38 | mdbooth | DISK_GB_PURCHASED_OCTOBER | |
| 13:42:56 | dansmith | obviously that would be a trait not a resource class :) | |
| 13:43:04 | mdbooth | Hehe | |
| 13:45:35 | ganso | mriedem: good morning! =) I replied to your comment in https://review.opendev.org/#/c/657870 Could you please confirm if the test case is really the same? | |
| 13:45:41 | kashyap | efried: Nice work on doing the summary emails. Tireless donkey work it is. | |
| 13:45:46 | kashyap | Thanks! | |
| 13:49:51 | efried | kashyap: :) thanks | |
| 13:52:16 | efried | shilpasd: afaict, the libvirt driver is setting the DISK_GB inventory based on the result from _get_local_gb_info() | |
| 13:53:04 | efried | At a glance, it seems like it would make sense to split that into two methods: _get_local_disk_info and _get_shared_disk_info. | |
| 13:53:41 | efried | If we start off supporting either local or shared but not both, then in a shared scenario the locals ought to report zero. | |
| 13:54:16 | efried | If the provider tree you are given has its disk local, and _get_local_disk_info returns zeros but _get_shared_disk_info returns nonzeros, you know you need to reshape. | |
| 13:54:53 | efried | And local_gb_used (really local_gb<anything>) should simply be fed from _get_local_disk_info, no matter what, meaning in the above case it will be zeros. | |
| 13:56:45 | stephenfin | anyone know if/when jaypipes will be back around. It sure would be lovely give the a once over and get it in (he reviewed in depth previously) https://review.opendev.org/#/c/629589/ | |
| 13:59:36 | jaypipes | stephenfin: I am here. | |
| 14:03:01 | jaypipes | stephenfin: +2 from me. feel free to +W it. | |
| 14:04:35 | shilpasd | efried: thanks for input | |
| 14:09:04 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Train Cycle Themes https://review.opendev.org/657171 | |
| 14:13:50 | ganso | Hi folks! I got a question about the num_instances value in the compute_nodes table. I have run into a situation where I have 4 VMs and "openstack hypervisor show" displays running_vms = 2. I found that this value is incremented/decremented in nova/compute/stats.py. Is there a way to recalculate this value or is the only way to fix this to update the database directly? | |
| 14:20:59 | mriedem | ganso: are all 4 of those vms running? | |
| 14:21:10 | ganso | mriedem: yes | |
| 14:23:13 | mriedem | looks like the update_available_resource periodic task in the compute should recalculate that value on each run of the task, which is every 1 minute by default | |
| 14:23:42 | mriedem | which release are you on? | |
| 14:24:42 | ganso | mriedem: mitaka | |
| 14:27:55 | mriedem | ganso: ok there was a stats related bug that probably only got fixed back to ocata, se | |
| 14:27:56 | mriedem | *sec | |
| 14:28:49 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Disable max_placement_results if affinity (or anti) is requested https://review.opendev.org/658110 | |
| 14:32:05 | mriedem | ganso: i think you might need https://review.opendev.org/#/q/I0b9e5b711878fa47ba90e43c0b41437b57cf8ef6 | |
| 14:32:30 | mriedem | oh but looking at that again, it was fixing a regression in ocata, so maybe not | |
| 14:32:33 | mriedem | nvm | |
| 14:32:37 | efried | tssurya: Was your intent to backport that ----^ | |
| 14:33:06 | tssurya | efried: yea was hoping to once it got merged in master | |
| 14:33:25 | efried | tssurya: Okay. Because we talked about enabling some placement-isms to help with this in Train itself, yah? | |
| 14:33:43 | tssurya | efried: yep, that would be a spec | |
| 14:33:53 | tssurya | I'll get onto that too soon-ish | |
| 14:33:54 | mriedem | in_tree for strict affinity but that's not backportable | |
| 14:33:58 | mriedem | it's also racy | |
| 14:34:19 | ganso | mriedem: so, this problem is likely caused by a bug that got fixed, correct? or was the periodic task added in ocata? | |
| 14:34:22 | efried | cool, just making sure I had the big picture | |
| 14:34:26 | tssurya | mriedem: no no ^ that fix is just making limits None if affinity/anti is requested | |
| 14:34:41 | tssurya | like we did earlier on for force_hosts/nodes | |
| 14:34:49 | tssurya | only that needs to be backported | |
| 14:34:49 | mriedem | ganso: the periodic was always around, | |
| 14:34:58 | mriedem | the stats thing i fixed was a regression in ocata so shouldn't affect you on mitaka | |
| 14:36:32 | mriedem | ganso: for that num_instances thing, you'd likely need to add some debug logging to that update_available_resource flow to see what it's doing, | |
| 14:36:58 | mriedem | but it should 0 out num_instances in the stats at the start of the run, and then add instances to the stats as it processes instances for that host/node | |
| 14:37:31 | mriedem | ganso: does https://developer.openstack.org/api-ref/compute/#list-hypervisor-servers also show 4 servers on that hypervisor? | |
| 14:38:16 | efried | tssurya: We have in_tree in Stein though... | |
| 14:38:51 | efried | ...so you could make a backportable patch that adds in_tree for positive affinity and removes the limit for anti-affinity. | |
| 14:39:01 | efried | ...and then if you want to backport further, use this one. | |
| 14:39:03 | ganso | mriedem: hmmm is that equivalent of "openstack hypervisor list" ? | |
| 14:39:09 | efried | tssurya: is that the plan? | |
| 14:39:35 | mriedem | efried: tssurya: it should be a 2 part series at least, | |
| 14:39:37 | ganso | mriedem: I have confirmed that the database has the value "2", so the database is wrong | |
| 14:39:41 | tssurya | efried: oh didn't realise in_tree was there in Stein since it was used only in train | |
| 14:39:52 | tssurya | but what you said makes sense | |
| 14:39:56 | mriedem | tssurya: but the bug goes back further than that | |
| 14:39:57 | efried | tssurya: I *think* it's in stein, sec... | |
| 14:40:06 | mriedem | in_tree usage in nova is new in train | |
| 14:40:08 | mriedem | from tetsuro | |