Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-13
15:43:43 gibi and I think it is pretty impossible to do properly due to what assumption the code currently makes
15:43:49 dansmith so I wouldn't want to spend a bunch of effort or cause a bunch of destabilization in the image backend to do so
15:43:57 dansmith gibi: right, hence my concern yesterday :)
15:44:18 gibi I needed to feel the pain to understand :)
15:44:25 dansmith :D
15:47:00 dansmith gibi: so, one thing I was wondering
15:47:03 gibi even if we start making allocation according to the images in the cache, the current code sometimes copy things over for conversions and that doubles the disk usage temporarly, or fail if no disk space for such operation
15:47:53 dansmith gibi: did you find somewhere that the DiskFilter considered the size of the image? Because even considering disk free space, you don't know if the target host will need room for the image (if it's not cached) PLUS the root, etc disks as well
15:48:22 dansmith because I don't think it really took that into account.. i.e. adding the flavor disks and the possible space required for a separate image
15:49:06 gibi DiskFilter used the disk_available_least value from the DB. There we count current available free space in the $instances_path (among other things)
15:49:21 dansmith gibi: yeah, and like I said yesterday if we start doing allocations for images, we have a lot of cases we need to cover, a lot of new potential needs for healing that data, etc.. it concerns me to make a decision to start doing that so quickly
15:49:37 gibi dansmith: ^^ agree
15:50:10 dansmith gibi: right, but the scheduler only looks to see if root+swap+ephemeral will fit in that space, but it may still fail because the compute node doesn't have the image cached already and thus need root+swap+ephemeral+image space to do the work
15:50:41 dansmith it's less of a problem, but it's similar
15:50:49 gibi correct (shit, another edge case)
15:52:08 dansmith so here's a couple of less impactful options:
15:52:56 dansmith 1. Each time we cache an image, or run the periodic, we generate a disk allocation for the compute node uuid which consumes inventory according to how much space the cache is using
15:53:04 dansmith 2. Same as above, but adjust the reserved amount
15:53:50 dansmith both cases need to consider the case where the _base is not on the same filesystem as the instances, but there's less synchronization involved, and we're not spraying tons of new allocations into place
15:54:34 dansmith we could also make it a workaround that you opt into in the short term to see how it goes, because cleanup from it would be much easier (just nuke one $cnuuid allocation)
15:55:18 dansmith and we could make compute node startup nuke that allocation if present and the workaround is disabled (or the self-correcting reserved amount)
15:55:54 gibi right, I would use allocation instead of reserved as allocation has a consumer attached
15:56:16 gibi reserved would be a sum of configured + detected
15:56:30 gibi which is math, that I dont like :)
15:56:31 dansmith hopefully glance gives us enough information to be able to increase that allocation before we start the download, so we know "oh sorry, inventory says we don't have room for this base image, so fail()"
15:56:52 dansmith I know it would, but I think reserved would be more obvious to an operator
15:57:00 dansmith even though it's a composite value
15:57:20 dansmith just because listing allocations are a bunch of meaningless-to-the-human UUIDs
15:57:46 dansmith I'm not arguing for that, I'm just saying there're benefits both ways
15:57:57 gibi I see. yes it is a tradeoff
15:58:37 dansmith I gotta get on a call
15:58:41 dansmith food for thought
15:59:42 gibi dansmith: thanks! I appreciate your help
16:00:40 gibi I've checked. glance give use the physical size of the image on the API. Which is good for ensuring we have still disk for the download
16:02:14 gibi also I think even if nova is converting a qcow2 glace image to raw locally (due to force_raw_images config) the resulting raw file is sparse
16:02:34 gibi but it might dependent on the host OS + file system support
16:26:39 gibi dansmith: if we start allocating / reserving the cache disk usage in placement then do we still need a pre-filter as well? For me it is OK to simply let the boot fail on the compute side if the extra DISK_GB resource for the cache cannot be allocated?
16:27:02 gibi s/?//
16:27:21 gibi anyhow documented your idea in the bug report
16:27:55 dansmith gibi: weighing aside, any compute with enough space for an instance but not enough for the instance+image will become a magnet for new builds, which will all fail
16:28:42 dansmith so, yeah, we can just pretend that's not worth solving, but it also sucks because avoiding that is what we're trying to solve with placement
16:31:22 gibi OK, I can imagine this as a two step solution then. First the allocation / reservation management in the compute then second the pre-filter that uses instance + image DISK_GB request but allocate only the instance disk for the instance_uuid in placement
16:32:52 dansmith perhaps
16:40:04 openstackgerrit melanie witt proposed openstack/nova master: DNM Try out running sphinx-build in parallel for releasenotes https://review.opendev.org/727429
16:48:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: WIP: allow disabling image cache for raw images https://review.opendev.org/727261
17:23:58 openstackgerrit melanie witt proposed openstack/nova master: DNM Try out running sphinx-build in parallel for releasenotes https://review.opendev.org/727429
19:05:50 openstackgerrit Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301
20:20:37 openstackgerrit melanie witt proposed openstack/nova master: DNM Try out running sphinx-build in parallel for releasenotes https://review.opendev.org/727429
20:54:18 openstackgerrit Sean McGinnis proposed openstack/nova master: DNM: Test making EM branch release notes static https://review.opendev.org/727875
23:15:54 melwitt gmann: do you understand why even after version bump we can get pep8 errors in gate? https://review.opendev.org/727589
23:18:10 gmann melwitt: it would be an error now as hacking min version bump will stop the new checks added in flake8 3.8.0 version.
23:18:28 gmann this one- https://review.opendev.org/#/c/727347/1
23:19:15 melwitt gmann: yeah but... (sorry) I thought bumping the version _stops_ the new checks from getting pulled in>
23:19:26 melwitt s/>/?/
23:19:45 gmann sorry *would not be an error
23:19:53 gmann missing *not*
23:20:31 melwitt ok, makes sense. so why second patch needed? is it nice to have for future flake8 or something>
23:20:36 melwitt gah I keep hitting >
23:21:10 gmann yeah for future but we will not be able to verify it or all error till we have new hacking pulling new checks.
23:21:47 melwitt yeah, ok.
23:21:51 gmann I am going to release the new hacking version 4.0.0 which will pull new checks and then in 727589 patch we will bump hacking version to so that we can verify the fix
23:22:38 gmann I mean fix + new hacking version bump in a single patch.
23:31:57 melwitt gmann: yeah makes sense. +2 on the 3.0.1 patch though I wondered if it's needed to avoid gate failures, would have thought we'd have a gate-failure bug around it. that is, it wasn't clear to me if the patch is needed to fix gate failure
23:34:24 gmann melwitt: I did nova patch earlier before i described the situation in other patches cmt msg like this- https://review.opendev.org/#/c/727576/
23:35:29 gmann melwitt: basically it will 1. fix local run where the latest fixed hacking 3.0.1 is not pulled automatically. in case of gate it is pulled as fresh installation. 2. it will protect if future flake8 3.9.0 version pull other new checks.
23:35:30 melwitt ah I see
23:35:55 melwitt oh ok so not a gate issue
23:36:04 melwitt cool thanks for explaining
23:36:12 melwitt I was confused before heh
23:37:05 gmann otherwise hacking before 3.0.1 is capping "flake8<4.0.0" so flake8 3.8.0 which ic current release and 3.9.0 future release both can be protected by bumping hacking to 3.0.1
23:37:22 melwitt right
23:37:59 melwitt yeah, so needed for future
23:39:46 gmann it is actually confusing as all inbuild deps actually. hacking does protect cap for flake8 and flake8 does protect cap for pycodestyle. but if pycodestyle is pulled as latest by some flake8 plugins then again it fail like flake8-import-order then we need pycodestyle cap explicitly. like tempest case- https://review.opendev.org/#/c/727276/2/test-requirements.txt
23:40:13 gmann so our way of handling the flake8 with plugins is not so great.
23:40:22 melwitt wow. mind blown
23:41:24 gmann i added PR on flake8-import-order but not sure if they will accept or not with our hacky use case :) - https://github.com/PyCQA/flake8-import-order/pull/172
23:41:56 melwitt ah, good luck :)
#openstack-nova - 2020-05-14
00:46:44 openstackgerrit Wenping Song proposed openstack/nova master: delete sub resource provider when delete resource provider https://review.opendev.org/719163
00:49:25 openstackgerrit melanie witt proposed openstack/nova master: Configure latex_engine = 'xelatex' for pdf docs build https://review.opendev.org/727898
01:07:14 openstackgerrit melanie witt proposed openstack/nova master: Configure latex_engine = 'xelatex' for pdf docs build https://review.opendev.org/727898
01:32:15 openstackgerrit Merged openstack/nova-specs master: Re-propose provider-config-file spec for Victoria https://review.opendev.org/725788
01:36:51 openstackgerrit Merged openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication' https://review.opendev.org/719720
02:48:28 tony_su gibi, sean-k-mooney : Thanks for reviewing and merging the spec.
02:50:43 tony_su gibi, sean-k-mooney : I will start to rebase related patches and send out for code review once they are ready.
03:03:55 melwitt fyi the openstack-tox-docs job is failing for nova since a recent upper-constraints bump, I have proposed a fix here https://review.opendev.org/727898 I have also sent a ML post
05:56:38 openstackgerrit melanie witt proposed openstack/nova master: Configure latex_engine = 'xelatex' for pdf docs build https://review.opendev.org/727898
07:10:38 openstackgerrit Nalini Varshney proposed openstack/nova master: Add migration to make key field type VARBINARY in aggregate_metadata table, https://review.opendev.org/725522
07:35:02 brinzhang_ sean-k-mooney: https://review.opendev.org/#/c/727589/2 do I need to update this patch? I have not concern stephenfin has already fixed these issue, but he abandoned.
07:49:09 gibi good morning Nova
07:49:29 bauzas whoops my manners
07:49:34 bauzas good morning Nova
07:50:26 brinzhang_ morning nova
07:50:59 brinzhang_ hi gibi, bauzas, I saw you are -1 for https://review.opendev.org/#/c/727140/1, the flake8 error fix patch
07:51:44 gibi brinzhang_: my -1 is about the fact that 3.8.1 added yet another check that needs to be fixed
07:51:48 brinzhang_ I submit a same one, before that I have not concern stephenfin's patch, do I need to continue ?
07:52:24 brinzhang_ gibi: I will see http://paste.openstack.org/show/793434/
07:52:40 bauzas brinzhang_: my concerns was about making sure what we would need to change
07:52:49 gibi I guess stephenfin abandoned the patch as we went a different direction, capping the version
07:53:34 brinzhang_ yeah
07:53:55 brinzhang_ I mean, if there is no sense to fix, I think I would abandon too.

Earlier   Later