| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-12 | |||
| 08:29:51 | huaqiang | how about let me re-arrange the order | |
| 08:30:40 | huaqiang | and resolving some dependencies with slight change to make it possible to be reviewed parallelly | |
| 08:32:18 | openstackgerrit | Nalini Varshney proposed openstack/nova master: Add migration to make key field type VARBINARY in aggregate_metadata table, https://review.opendev.org/725522 | |
| 08:35:00 | bauzas | gibi: impressive count on open bugs https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New | |
| 08:35:12 | bauzas | gibi: how have you triaged them ? | |
| 08:48:12 | gibi | bauzas: most of the did not have enough information | |
| 08:48:50 | gibi | bauzas: and during the last two weeks we get only about 5 new bug reports so the low inflow helped a lot | |
| 08:49:51 | gibi | sean-k-mooney: thanks, I will check it | |
| 08:50:42 | gibi | jkulik: what you want to achive? Do you want to move users from old HW gen to new HW gen over time? | |
| 08:51:07 | stephenfin | huaqiang: Sure. The only issue with that is that future rebases are harder. If you can keep the existing +2s though, go for it | |
| 08:52:29 | jkulik | gibi, we have different hardware versions in place at the same time, where, because of NUMA and whatnot, flavors differ slightly. We want a user to be able to deploy a "2 TB instance" flavor, even if that means 1.8 on one node and 2.1 on the other. | |
| 08:53:35 | sean-k-mooney | jkulik: that is how it works by default | |
| 08:54:18 | sean-k-mooney | unless you meen 1.8 TB on one and 2.1 TB on the other? in not sure what those number refer too | |
| 08:54:48 | jkulik | sean-k-mooney, the flavor has to map to 1.8 TB on one hardware version and 2.1 TB on the other | |
| 08:55:00 | sean-k-mooney | jkulik: ya that is not allowed | |
| 08:55:01 | jkulik | sean-k-mooney, we just want our users to be able to specify "give me that 2 TB thingy" | |
| 08:55:22 | sean-k-mooney | right but its a 1.8TB thing and a 2.1TB thing | |
| 08:55:31 | sean-k-mooney | its not a 2TB thing in either case | |
| 08:55:43 | jkulik | because it's hard for us to keep hypervisors empty enough to support that size, the user can't really know up-front what hardware is currently free | |
| 08:55:47 | sean-k-mooney | you could use boot form volume and not provide any default root disk | |
| 08:56:09 | jkulik | we're talking RAM here :/ | |
| 08:56:20 | sean-k-mooney | oh i taught you ment disk | |
| 08:56:33 | sean-k-mooney | and the user should know know up front how much ram is free | |
| 08:56:38 | sean-k-mooney | *should not | |
| 08:57:09 | sean-k-mooney | jkulik: it sound like you want to reconfigure the scheduler to pack | |
| 08:57:10 | openstackgerrit | Tony Su proposed openstack/nova-specs master: Re-propose provider-config-file spec for Victoria https://review.opendev.org/725788 | |
| 08:57:16 | jkulik | I don't see how we could tell the user what flavors could currently get deployed via nova's api | |
| 08:57:39 | sean-k-mooney | if you configure the schduler to pack based on ram usage it will try to keep hosts free if it will fit on othet hosts | |
| 08:57:57 | sean-k-mooney | jkulik: well you cant and that is kind fo intentional | |
| 08:57:59 | jkulik | sean-k-mooney, the problem is even bigger for us: we use vmware .. so we have a cluster in the compute node and have to do some magic behind the scenes to convince vmware to spawn this monster | |
| 08:58:20 | sean-k-mooney | it breaks the cloud model if the tenant has a view of the aviaable capacity | |
| 08:58:47 | jkulik | yeah, that's why I wanted a flavor, that can match any of those free hosts, by having sub-flavors ;) | |
| 08:59:00 | sean-k-mooney | jkulik: yep which is not going to happen | |
| 08:59:12 | sean-k-mooney | we kind of had something like that at one point | |
| 08:59:14 | sean-k-mooney | for ironic | |
| 08:59:23 | sean-k-mooney | but that has been gone for a long time | |
| 08:59:57 | jkulik | we'll have to work something out downstream then, thanks for clarifying :) | |
| 09:00:24 | sean-k-mooney | im trying to think if we could use placmenet and custom resouces classes for this | |
| 09:00:48 | sean-k-mooney | but what you are trying to do is going against the grain of how nova is intended to be used ot a degree | |
| 09:00:52 | bauzas | gibi: ack thanks | |
| 09:03:05 | sean-k-mooney | jkulik: im not sure how configurable the vmware dirver is | |
| 09:03:30 | sean-k-mooney | but do you want to reserve space for this giant vms or no? | |
| 09:04:13 | openstack | Launchpad bug 1838309 in OpenStack Compute (nova) "Live migration might fail when run after revert of previous live migration" [Undecided,Won't fix] - Assigned to Vladyslav Drok (vdrok) | |
| 09:04:13 | bauzas | gibi: agreed with Wontfix https://bugs.launchpad.net/nova/+bug/1838309 ? | |
| 09:04:57 | sean-k-mooney | what you could do woudl be to reserve ram on the specifci hosts where you intend to spawn these isntances. then create a custom resouce class for them and a flavor that request 2TB or ram but set resocues:memory_mb=0 resouces:CUSTOM_GIANT_VM=1 | |
| 09:05:00 | jkulik | sean-k-mooney, not in general. we currently run something, that frees up enough space on one note - so we can spawn one. | |
| 09:05:32 | sean-k-mooney | ok so that wont work if you want it to be dynamic then | |
| 09:05:33 | jkulik | yeah, we have something like that. but it has to fit into normal quota for billing | |
| 09:05:47 | tony_su | gibi: This is Tony Su and I am doing provider-config-file re-propose thing. Could you kindly spare some time to review its spec (only minor changes in History and Assignee sections vs. Ussuri version). https://review.opendev.org/#/c/725788/ | |
| 09:06:22 | sean-k-mooney | jkulik: well the flavor.ram value would still be 2TB | |
| 09:06:32 | jkulik | sean-k-mooney, we dynamically create a sharing child-resource-provider for a node, that has the CUSTOM_GIANT_VM resource | |
| 09:07:23 | sean-k-mooney | ok but the issue is you still need to aling the flavor.ram to what that means for that host | |
| 09:07:24 | jkulik | sean-k-mooney, so currently, it works for us. but the flavor's RAM doesn't align with NUMA on all nodes. meaning, we would have to create more flavors. | |
| 09:07:31 | sean-k-mooney | based on the hardware version | |
| 09:07:35 | jkulik | yes | |
| 09:08:12 | sean-k-mooney | jkulik: may i ask why you actully use VM instead of ironic nodes at that scale | |
| 09:08:19 | jkulik | but given that we only free up one node, that could fit 2 TB either because it's 2 TB in size or because it's bigger, the 2 TB are more a rounded value the custerom might request | |
| 09:08:32 | tony_su | gibi: Really appreciate your support and help. | |
| 09:08:58 | jkulik | sean-k-mooney, sure. we were on ironic, but our customers would like to have the automatic failover when a node goes down, that vmware provides. also TCO. | |
| 09:10:25 | sean-k-mooney | i see so they are relying on the plathform for failover rahter then using an orchestration layer like k8s to mage there application | |
| 09:10:39 | ikla | the controller and node doesn't need to have the same device in in for passthrough? | |
| 09:10:40 | sean-k-mooney | so this is very much a pets not cattel usecase | |
| 09:10:50 | jkulik | since the customer can't find out, which precise flavor she can currently deploy via API, it would have been nice to have specific sub-flavors for ~ 2 TB | |
| 09:10:52 | gibi | bauzas, stephenfin: easy +2 https://review.opendev.org/#/c/721278 (ussuri spec move) | |
| 09:10:57 | ikla | my flavor keeps failing when trying to turn on a instance | |
| 09:11:03 | sean-k-mooney | ikla: the alias need to be on the computes and contoler | |
| 09:11:14 | jkulik | sean-k-mooney, yes. definitely pets | |
| 09:11:18 | sean-k-mooney | ikla: the pci whitelist is only needed on the compute | |
| 09:11:27 | ikla | alias yes, device only on compute.. | |
| 09:11:28 | ikla | k | |
| 09:11:52 | sean-k-mooney | ikla: so the contole just uses it to translate the value in the flavor into pci request for schduling | |
| 09:12:20 | openstack | Launchpad bug 1838309 in OpenStack Compute (nova) "Live migration might fail when run after revert of previous live migration" [Undecided,Won't fix] - Assigned to Vladyslav Drok (vdrok) | |
| 09:12:20 | gibi | bauzas: I agree with WontFix https://bugs.launchpad.net/nova/+bug/1838309 | |
| 09:12:29 | bauzas | cool, moving on | |
| 09:13:03 | gibi | tony_su: thanks for taking that feature over. I will look at the spec soon | |
| 09:15:27 | openstack | Launchpad bug 1878024 in OpenStack Compute (nova) "disk usage of the nova image cache is not counted as used disk space" [Undecided,Confirmed] | |
| 09:15:27 | gibi | bauzas: thanks for looking at the new bugs. You have the bug lock, I have yet another downstream originated issue to look at https://bugs.launchpad.net/nova/+bug/1878024 | |
| 09:16:12 | sean-k-mooney | gibi: thats a tricky one | |
| 09:16:40 | sean-k-mooney | in that really we should never fail to boot a vm becasue of a cached image taking up space | |
| 09:17:10 | sean-k-mooney | i.e. i would expect as an operator for the cache to be perurged first | |
| 09:17:25 | sean-k-mooney | so im not sure i would want it to be counted as used | |
| 09:17:47 | sean-k-mooney | that siad we dont purge the cache today as far as i know at least not while the image is used on the host | |
| 09:17:54 | sean-k-mooney | so i can see this causeing issues too | |
| 09:18:18 | sean-k-mooney | could you just workaround this using the host reserved disk paramater | |
| 09:18:47 | sean-k-mooney | or perhaps we need a config option to limit the size of the cache | |
| 09:19:37 | gibi | sean-k-mooney: the cache is used as a backing file for qcow root fs so it cannot be really purged | |
| 09:19:40 | sean-k-mooney | the easy option is just to add the size of the cache to the reserved value in placment but that will reduce the amount of vms that can be spawned | |
| 09:19:54 | sean-k-mooney | gibi: i tought we had two copies | |
| 09:20:01 | sean-k-mooney | on in the cache and a second for the backing file | |
| 09:20:20 | sean-k-mooney | isnt the cach module indpentend of the image backend | |
| 09:20:22 | gibi | sean-k-mooney: reserved_host_disk_mb would only be useful if we could limit the size of the cache | |
| 09:21:02 | sean-k-mooney | yep the two line "fix" is to jsut count the cache size and added it to the reserved value in placment | |
| 09:21:10 | sean-k-mooney | but i dont think that is a resonable long term fix | |
| 09:22:45 | gibi | simply counting it as reserved does not help when a VM is booted with a new image to a compute. There we would need to make sure that both the new cached image and the VM root fs fits the compute | |
| 09:22:58 | gibi | we discussed these options with dansmith yesterday | |
| 09:23:09 | gibi | see the summary in the bug and the link to the IRC discussion | |
| 09:24:14 | openstackgerrit | Merged openstack/nova-specs master: move implemented specs in ussuri https://review.opendev.org/721278 | |
| 09:25:35 | sean-k-mooney | gibi: are we sure we use the cache image directly for the backing file by the way | |
| 09:26:13 | sean-k-mooney | gibi: i tought that was how it worked but i remeber someone telling me its not how it worked and that we chage two copies one in the caceh and one for the backing file | |
| 09:26:59 | gibi | let me double check | |