Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-11
15:21:30 dansmith but yes, the scheduler is never going to know whether or not to pick a host based on cache availability, so you always have that
15:22:24 dansmith we could go totally crazy and create an allocation for each image, by image uuid and after selecting a host, the scheduler could check to see if there was an allocation for that image against the host's provider to decide if it thinks it will fit :)
15:22:43 dansmith that has some nice benefits, but it's a little crazy and there's still plenty of room for racing of course,
15:23:03 dansmith and plenty of room for exhausting all the candidates in a small query set, leading to non-ideal looping of retries
15:23:39 dansmith plus we'd have to have a separate provider for the cache disk if they're separate
15:27:07 gibi complicated indeed
15:27:41 gibi I will pass the workaround of having the cache on a different partition to downstream
15:27:56 gibi at least that is something that the downstream project can do
15:28:24 aarents gibi: dansmith we have this issue, painfull one
15:28:48 dansmith gibi: ack
15:29:02 aarents gibi yep in some case we put cahe in another file system to get ride of this
15:29:22 gibi dansmith: the old DiskFilter had the disk_available_least info to prevent overallocation but we removed the DiskFilter
15:30:27 dansmith yeah, fair point
15:30:56 gibi we might want to re-introduce something like diks_available_least as a filter? or a pre-filter with placement support?
15:31:40 dansmith I think diskfilter had plenty of other problems, like the other way where the filter behavior conflicted with the hypervisors listing, which definitely causes support cases
15:31:50 dansmith not that they agree now, but..
15:32:18 dansmith gibi: I don't think we'd want that to be a pre-filter because you'd have to provide either an inclusion or exclusion list of all hosts to placement each time
15:32:40 dansmith not like a trait or aggregate, but "any one of these hosts: [... array of 5000 ...]
15:34:00 gibi yeah I don't want to bring back the whole DiskFilter, just bring back the extra information to the scheduler / placement that how much actual disk space is free under the instance_path
15:34:30 gibi as an idea
15:34:59 dansmith gibi: all I'm saying is doing it as a pre-filter is the wrong place
15:35:06 gibi ack
15:35:28 dansmith we might still be reporting that value such that a filter can check it
15:38:17 dansmith one other option is that the cache_images() thing that I added is setup to work as a call, returning information about presence,
15:38:44 dansmith so if we were to pass it a "don't download just check" flag, or a "download in background" flag, then we would get back an indication if it's present or not,
15:39:04 dansmith which would let the scheduler know whether or not to filter out hosts with 1x or 2x the disk space
15:40:00 dansmith that's pretty heavy, and would likely need to be done from conductor,
15:40:21 dansmith but it's a little less racy than checking some minutes-old disk free amount and assuming it's going to work
15:40:54 dansmith another cloudy way to look at this is to say we should just have people weigh hosts on free space,
15:41:07 dansmith in line with our "we don't schedule at capacity" project goal
15:42:35 gibi dansmith: so if the rpc call reports that the image is not cached, then we would add 2x disk space the allocation candidate query, but only allocate 1x disk space on the selected host for the instance, then on the compute side the image cache code would allocate the other 1x disk space in placement for the cache
15:42:42 dansmith no
15:42:48 dansmith we've already done the a-c query at that point
15:43:02 dansmith we'd just use that to advise us which of the a-c are valid
15:43:11 gibi ahh yeah, you have to now which host you send the rpc call
15:43:14 dansmith right
15:46:14 gibi the weigher thing is good for big deployments but will fall short for edge. As far as I understand my downstream report is from a really small edge site close to capacity. :/
15:47:40 dansmith indeed, although I think I'd argue that for highly constrained situations the separate filesystem is the right approach there anyway, given the (a) usually constrain-able image sets for edge and (b) the need to avoid the race that we'll have in some form anyway
15:47:46 dansmith but yep, it's not a great answer for someone that just wants it to work ideally
15:48:29 gibi I will definitely suggest the separate partition for now as I feel whathever solution we come up with (if any) it will not be backportable
15:48:38 dansmith for sure
15:49:53 dansmith fwiw, making a pre-call to cache_images(background=True) would theoretically give us some lower time-to-boot performance in other cases
15:50:10 dansmith I'm really not sure whether that's a terrible idea or not, but it's an interesting thought
15:50:30 dansmith similar to the cyborg case of starting the programming at bind time from the conductor in parallel to the rest of the instance bringup
15:50:36 gibi it would have a side effect to cache image to a compute that otherwise will not be selected
15:51:18 gibi hm or not
15:51:31 dansmith in the tight case you mean right? We'd start caching an image on a host that the scheduler was going to exclude anyway, which is true
15:52:11 gibi yeah for the thight case when the image would fit into the cache but the instance root disk would not any more
15:52:16 dansmith yep, for sure
15:52:24 gibi but that really tight
15:53:54 gibi what if for thight cases we allow disabling the cache entirely? it is thight so no space for cache
15:54:30 dansmith doing that would require a substantial resdesign of the whole image backend for libvirt I think
15:55:03 gibi is it because we assume that there is a backing file for the root fs image which happen to be the cached image?
15:56:41 dansmith I'm actually not sure what happens if you configure qcow2 and "flatten_images" actually, I'd have to look
15:56:50 dansmith that might have the same effect, I'm not sure
15:59:21 gibi do you mean force_raw_images conf option?
16:00:26 gibi or use_cow_images = False? or a linear combination of the two :)
16:00:44 markguz_ Hi nova folks. i have an instance that failed that i migrated (not live) to another compute host, but upon restarting it errors with
16:00:45 markguz_ Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed'
16:00:58 dansmith gibi: yeah there's some combination that results in full flattening, but I'm not sure what they are
16:01:10 dansmith gibi: not sure if tat actually results in the cache image going away, or getting copied or what
16:01:24 dansmith actually, as to be the latter I think since we have to expand the size of it
16:01:26 markguz_ i've tried the various solutions found via google but non work
16:01:46 dansmith so you probably end up with 2x the space initially at least, and then you'd need to immediately purge the original or something
16:02:22 gibi dansmith: ack
16:05:47 gibi dansmith: hm even when the flat backend copies the image to raw it does update the cache to keep the base image https://github.com/openstack/nova/blob/d6450879c7f7dd19366b6f002301fbbf87918026/nova/virt/libvirt/imagebackend.py#L585
16:08:05 gibi dansmith: anyhow thanks for your thoughts I have to drop today soon so I will add some summary of this discussion to the bug.
16:10:06 dansmith gibi: yeah, that's what I was thinking above when I said "but then you'd need to immediately purge"
16:10:16 dansmith gibi: the cache is trying to be a cache
16:12:45 gibi yeah, I see now
16:23:26 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898
16:30:14 gibi lyarwood, dansmith: updated the bug 1878024 with what we talked about above.
16:30:14 openstack bug 1878024 in OpenStack Compute (nova) "disk usage of the nova image cache is not counted as used disk space" [Undecided,New] https://launchpad.net/bugs/1878024
16:30:24 dansmith cool
16:30:57 gibi and now I go and bake some bread for dinner
16:31:04 gibi see you tomorrow
18:15:35 openstackgerrit Merged openstack/nova master: Support for --force flag for nova-manage placement heal_allocations command https://review.opendev.org/715395
18:15:48 openstackgerrit Merged openstack/nova stable/queens: Include only required fields in ironic node cache https://review.opendev.org/724862
18:16:01 openstackgerrit Merged openstack/nova stable/queens: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723054
18:16:10 openstackgerrit Merged openstack/nova stable/rocky: Add config option for neutron client retries https://review.opendev.org/722819
18:16:27 openstackgerrit Merged openstack/nova master: Suppress remaining policy warnings in unit tests https://review.opendev.org/726272
18:22:04 markguz_ anyone know how to get out of Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed' hell?
19:35:26 openstackgerrit Harshavardhan Metla proposed openstack/nova master: [Nova] Add reference to Placement installation guide https://review.opendev.org/726936
19:42:27 markguz_ for anyone that's interested setting neutron.ml2_port_bindings.vif_type to "ovs" in the database fixed this for me
19:43:35 sean-k-mooney markguz_: if you get that error its because neutron failed to bind the port
19:43:47 markguz_ i know
19:43:50 sean-k-mooney which normally means there was an error on cthe compute node
19:44:10 markguz_ but once it happens it seems next to impossible to fix it via the normal methods.
19:44:46 sean-k-mooney markguz_: you fix it by setting the host filed to "" or "none" then back to the original hostname
19:45:09 markguz_ the instance will not boot due the "binding_failed" being written into the vif_type field
19:45:33 markguz_ sean-k-mooney: or by updating that field i just mentioned in the neutron db
19:46:02 sean-k-mooney yes but you can do "openstack --os-cloud=admin port set --host none baf2b165-797b-4305-bc6b-5b63250b890d" follow by " openstack --os-cloud=admin port set --host workstation baf2b165-797b-4305-bc6b-5b63250b890d"
19:46:09 sean-k-mooney to do it from the api without db hacking
19:46:36 sean-k-mooney it will actully cause port binding to happen properly recalualting the correct values
19:46:41 markguz_ sean-k-mooney: ok. thanks for that
19:47:01 sean-k-mooney so i did that yesterday becauses i was swaping form the iptables firewall dirver to the ovs one
19:48:18 markguz_ this happened to me when i did a non-live migrate of a shutdown instance to a new host
19:48:42 sean-k-mooney hum it should not happen in that case
19:49:10 sean-k-mooney something obviosly went wrong there should be an error in the neutron server log
19:49:46 markguz_ yeah. haven't had time to deep dive. Was focussed on getting the instance back online

Earlier   Later