| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-22 | |||
| 17:31:48 | dansmith | EugenMayer: yeah, not helping :) | |
| 17:32:00 | EugenMayer | dansmith sorry. my bad. | |
| 17:32:11 | sean-k-mooney | where they allowed you to update a flavor by deletign and recating it but ya lets not talk about that issue now | |
| 17:35:16 | EugenMayer | well if you ask me to the state error - one should not mark the instance as 'error' if a image_backup task failed - there is no reason for that. Creating a glance image does not required the instance to shutdown or similar, this said, i assume both task (the instance running) and the creation of the image can work in parallel and are independent | |
| 17:35:54 | dansmith | EugenMayer: going to error state is just the nova convention (in most places) | |
| 17:36:00 | EugenMayer | so this said, if the image_backup task is: failed, the task_id does no longer exists or whatever, nova should not block 'restarting the instance' | |
| 17:36:11 | dansmith | and if the issue wasn't critical, then a start operation will clear the error state without requiring a reboot of the actual instance | |
| 17:36:16 | EugenMayer | dansmith well it is the 'better safe then sorry convention i guess' | |
| 17:37:09 | dansmith | EugenMayer: we're agreeing with you that we do not expect that this is something that should be so jammed up and that there's probably some missing error handling in this case | |
| 17:37:31 | dansmith | I'm describing what the usual nova error procedure is, regarding going to error state to signal to the user that their thing didn't happen | |
| 17:37:35 | dansmith | it's not great, it's just the convention | |
| 17:37:44 | sean-k-mooney | EugenMayer: creatign the glance image might require the instnace to be shutdown by the way | |
| 17:37:47 | dansmith | because if you do a backup, and the instance goes to active, you assume it worked, but it didn't | |
| 17:37:58 | sean-k-mooney | snapshots are not guareentee to be live | |
| 17:38:01 | dansmith | right | |
| 17:38:07 | EugenMayer | if nova is the task owner, which i understood is the case, it should design a propere state machine in case the task (which i understood is blocking via REST, so very fragile). Task could complete failed or succeeded. Task could never complete or even be deleted (on the glance side) | |
| 17:38:43 | sean-k-mooney | EugenMayer: there was a effort to do that at one point but this is also a distibuted system problem | |
| 17:38:49 | dansmith | EugenMayer: there's no task | |
| 17:38:49 | EugenMayer | understood, but i assume the sequence is: shutdown/sleep instance, create snapshot, start/resume instance, upload snapshot to glance .. (do task tracking) | |
| 17:39:42 | sean-k-mooney | EugenMayer:right yes but there may be clean up to be done in the compute node or stoage backend if the upload fails | |
| 17:40:20 | EugenMayer | no task means: it's blocking only. Understood there is no task_id or somewhat, just a blocking http-call. So as you both suggested, this blocking call needs to cleanup in all cases: 200,500 and also 408 and others. | |
| 17:40:22 | sean-k-mooney | such as deleting the file we created that was not uploaded | |
| 17:41:05 | dansmith | EugenMayer: we're saying exactly that.. we should, assuming we can | |
| 17:41:11 | sean-k-mooney | EugenMayer: yep and nova shoudl check the respocne code and start cleaning up if it failed | |
| 17:41:12 | EugenMayer | the i have seen the glance image task under image, which i was able to delete, but since the blocking request disconnected far ago, no cleanup happened on the nova side | |
| 17:41:28 | dansmith | EugenMayer: there are cases that are more complicated, such as with ceph where we might not be able to recover at all, depending on what happened, but in general we agree | |
| 17:41:41 | EugenMayer | agreed | |
| 17:42:16 | sean-k-mooney | well recovery in ceph might be squash/merge the ceph snapshot back into the previous volume for example | |
| 17:42:36 | dansmith | depends on the failure of course | |
| 17:42:54 | sean-k-mooney | where as for qcow we woudl mirror the file on disk then upload and if it faile delete the copy | |
| 17:43:45 | sean-k-mooney | EugenMayer: if you have logs and or a repoduce please file a bug and we can see if we can figure out why nova is not cleaning up as expected | |
| 17:52:07 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |
| 17:52:08 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 17:52:08 | opendevreview | Stephen Finucane proposed openstack/nova master: trivial: Clean manager.Manager, service.Service signatures https://review.opendev.org/c/openstack/nova/+/764806 | |
| 17:58:03 | EugenMayer | sean-k-mooney dansmith will do, thank you for your both time | |
| 18:22:04 | admin1 | hi all .. i am hitting this bug, https://bugs.launchpad.net/glance/+bug/1916482 , but don't have an idea on how to solve it .. i am using openstack-ansible and the latest tag 24.0.1 | |
| 18:22:12 | admin1 | nova is local disk, glance is rbd | |
| 18:22:38 | opendevreview | Stephen Finucane proposed openstack/nova master: objects: Remove unnecessary type aliases, exceptions https://review.opendev.org/c/openstack/nova/+/738240 | |
| 18:22:39 | opendevreview | Stephen Finucane proposed openstack/nova master: objects: Use imports instead of type aliases https://review.opendev.org/c/openstack/nova/+/738018 | |
| 18:22:39 | opendevreview | Stephen Finucane proposed openstack/nova master: objects: Remove wrappers around ovo mixins https://review.opendev.org/c/openstack/nova/+/738019 | |
| 18:22:40 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: add ovo-mypy-plugin to type hinting o.vos https://review.opendev.org/c/openstack/nova/+/758851 | |
| 18:36:31 | opendevreview | Ghanshyam proposed openstack/nova stable/xena: DNM: testing centos8 py36 job https://review.opendev.org/c/openstack/nova/+/834765 | |
| 18:38:10 | opendevreview | Ghanshyam proposed openstack/nova stable/wallaby: DNM: testing centos8 py36 job https://review.opendev.org/c/openstack/nova/+/834721 | |
| #openstack-nova - 2022-03-23 | |||
| 00:16:37 | EugenMayer | is it possible to somehow manually | |
| 00:16:52 | EugenMayer | (database manipulation) set the flavor id of an instance? | |
| 00:38:47 | opendevreview | anguoming proposed openstack/nova master: fix the bug of the log line has no request_id info at source host when live migration https://review.opendev.org/c/openstack/nova/+/834677 | |
| 00:38:51 | EugenMayer | well i just created a new flavor of the same (old size) and then changed the flavorid in the nova_api database to match the expected old flavor id. Interestingly i could not find the relation table from flavor<->instance. it's not in nova::instances nor in nova_api:: anyway - where is this relation stored? | |
| 00:39:40 | sean-k-mooney[m] | its not we embed a copy of the flavor in the instance_extra table | |
| 00:40:17 | sean-k-mooney[m] | i belive we store teh orginal flavor name in the instnace in the api db | |
| 00:41:02 | sean-k-mooney[m] | but the only supported way to change the flaovor is via a resize | |
| 00:41:15 | EugenMayer | well you cannot do that if terraform bricked it | |
| 00:41:38 | EugenMayer | if a flav has been deleted while still the relation exists, which happened, you can neither resize nor do anything with this instance | |
| 00:41:39 | sean-k-mooney[m] | you should be able to do it from the api | |
| 00:41:47 | EugenMayer | no it wont work - 409 | |
| 00:42:06 | EugenMayer | i tried it all via api before touching the database manually - for the obvious reasons | |
| 00:42:28 | sean-k-mooney[m] | you should be able to resize the instnace even if the flavor has been deleted | |
| 00:42:50 | EugenMayer | openstack server resize --flavor <newid> <serverid> will end up throwing a 409 if the current flavorid, which is referenced, does not exist | |
| 00:43:04 | EugenMayer | > you should be able to resize the instnace even if the flavor has been deleted | |
| 00:43:11 | sean-k-mooney[m] | hum that sound like a bug | |
| 00:43:13 | EugenMayer | this is not possible, i just did try exactly that. | |
| 00:44:59 | sean-k-mooney[m] | i dont knwo why we would need the old flavor to exist | |
| 00:45:32 | EugenMayer | thank you for the hint with instance_extra - that would be the better way to change the flavor there. I'am aware that the flavor must match the current instance specs if i do it on the db layer - all i want is fixing the actual relation so the API is operating again | |
| 00:46:26 | EugenMayer | it does not need it, it will just try to load the entity and resolve all it's relation and as a side effect an exception will be thrown, that the flavor entity cannot be loaded | |
| 00:47:03 | EugenMayer | and that exception most probably bubbles up and ends up canceling the request | |
| 00:47:28 | sean-k-mooney[m] | the request spec has referneces to the flavor too in the api db | |
| 00:48:42 | EugenMayer | i really see that the entire 'distributed system' and microservice architecture introduced a lot of issue here. We have a database nova_api defining ::flavor. It's PK is used as in FK in nova::instance_extra but without any constraint or anything else - since the databases 'are seperated' - which they are not at all. | |
| 00:49:19 | sean-k-mooney[m] | they are we store copies of the full flavor | |
| 00:49:35 | sean-k-mooney[m] | so the request spec has the serisalsed flavor embeded in it | |
| 00:49:44 | sean-k-mooney[m] | which is used for schduling | |
| 00:50:37 | sean-k-mooney[m] | nova is not a collection of microservice by the way | |
| 00:50:57 | sean-k-mooney[m] | its a single distbuted service with multiple components that work together | |
| 00:52:50 | EugenMayer | why did the databases have been designed this way? AFAICS there are relations from one entity in one database to a different database - this really is not how this should be designed right. so nova::instance_extra.flavor<->nova_api::flavor.flavorid | |
| 00:53:20 | sean-k-mooney[m] | that is not how it works | |
| 00:53:42 | sean-k-mooney[m] | when you boot an instance we make a copy of the flavor and store a copy in the instance table | |
| 00:53:55 | EugenMayer | in which instance table: | |
| 00:53:57 | sean-k-mooney[m] | flavor are imuntable and cant be change once created | |
| 00:54:05 | sean-k-mooney[m] | however flavor extra specs are mutable | |
| 00:54:06 | EugenMayer | nova::instances ? | |
| 00:54:28 | sean-k-mooney[m] | so we have to copy the flavor to make sure existing instnace are not change if you modify the extra specs | |
| 00:55:26 | EugenMayer | i guess you the then reference nova::instances.vcpus/memory_mb/disk_db and so on | |
| 00:56:20 | sean-k-mooney[m] | yes those are fixed and cannot be modified. | |
| 00:56:35 | EugenMayer | while i understand that flavors are immutable (and should be) and the actual fields are copied / flatted into nova::instances -there is still a relation from nova::instance_extra to nova_api:falvor | |
| 00:57:00 | sean-k-mooney[m] | yes and no | |
| 00:57:16 | EugenMayer | and if this relation is broken, which very well can happen since there is no constraint possible - you can no longer do anything with the instance. You can no longer apply any new flavor or resize it | |
| 00:57:49 | EugenMayer | so entire decoupling has been planned maybe, but it seems not have been implemented (yet) | |
| 00:57:52 | sean-k-mooney[m] | you shoudl be able to today | |
| 00:57:55 | sean-k-mooney[m] | if you cant its a bug | |
| 00:58:17 | EugenMayer | well then there is a bug in xena (not sure what today standard is, yoga or xena?) | |
| 00:58:53 | sean-k-mooney[m] | so you should be aware that flavors used to be defiend at the cell level with cells v1 | |
| 00:59:31 | sean-k-mooney[m] | and when we moved to cells v2 we change the relation ships such that flavor were defiend gloablly in the api db | |
| 00:59:41 | sean-k-mooney[m] | but we conintued to keep the copy in the cell db | |
| 01:00:28 | sean-k-mooney[m] | but the intnace has used the copy in the request spec for schduling and instance extra as the source of truth for a vm for many many years | |
| 01:01:09 | EugenMayer | i understand, but there seems to be some usage of the back-reference anyway - maybe unintended | |
| 01:02:33 | sean-k-mooney[m] | so when we do a resize this is where we get the current flavor https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4127 | |
| 01:04:48 | EugenMayer | it seems, nevertheless, not the right design. nova should never include the actual flavorid for an instance, rathere nova_api should hold the reference from instance_uuid to flavorid - the relation owner here is nova_api. An instance is (as you explain) able to live without it's flavor - sureley a flavor can live without an instance too. But not | |
| 01:04:48 | EugenMayer | the relation. Sure the relation should be deleted in either way (deletion of instance or deletion of flavor). Anyway | |
| 01:05:32 | sean-k-mooney[m] | well you talking about something that has been in place for the better part of the last decade | |
| 01:05:36 | EugenMayer | looking at the code, nothing check for current_flavor to be null or not, it is used below right away. So if it is null / cannot be loaded (and not loaded does not trigger an exception already) this code will go bananas anway | |
| 01:05:52 | sean-k-mooney[m] | we have desgin constratint the mean we cant eaisly change the db schema | |
| 01:06:09 | sean-k-mooney[m] | espically for some of our larger deploymnets | |