Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-03-22
17:30:01 EugenMayer i did not yet check the tf openstack provider implementation to see what they have implemented and how that is a timing issue in the first place (since it does not happen every timee) .. but if i look at the openstack rest api / nova api .. swapping flavors is not designed at all.
17:30:19 sean-k-mooney EugenMayer: well flavor are intenede to be imuatble so you idealy woudl not delete them until all instance using them are resized
17:30:28 sean-k-mooney we do cache the flavor
17:30:30 sean-k-mooney in the insntace
17:30:42 dansmith EugenMayer: are you describing two issues or one? if the former, then let's not complicate diagnosing this one
17:30:47 sean-k-mooney but really you shoudl try to avoid removing flavor or image that are in use
17:31:08 EugenMayer well i cannot tell why tf openstack providere deletes the flav too early or whatever happens in detail (i did not check the sequence in the code yet)
17:31:23 sean-k-mooney EugenMayer: it should not delete it at all
17:31:31 EugenMayer dansmith sorry, my bad. second issue (the latter one with the flav)
17:31:44 sean-k-mooney it sould like they are implementing the hacky workflow that horizon use to have
17:31:48 dansmith EugenMayer: yeah, not helping :)
17:32:00 EugenMayer dansmith sorry. my bad.
17:32:11 sean-k-mooney where they allowed you to update a flavor by deletign and recating it but ya lets not talk about that issue now
17:35:16 EugenMayer well if you ask me to the state error - one should not mark the instance as 'error' if a image_backup task failed - there is no reason for that. Creating a glance image does not required the instance to shutdown or similar, this said, i assume both task (the instance running) and the creation of the image can work in parallel and are independent
17:35:54 dansmith EugenMayer: going to error state is just the nova convention (in most places)
17:36:00 EugenMayer so this said, if the image_backup task is: failed, the task_id does no longer exists or whatever, nova should not block 'restarting the instance'
17:36:11 dansmith and if the issue wasn't critical, then a start operation will clear the error state without requiring a reboot of the actual instance
17:36:16 EugenMayer dansmith well it is the 'better safe then sorry convention i guess'
17:37:09 dansmith EugenMayer: we're agreeing with you that we do not expect that this is something that should be so jammed up and that there's probably some missing error handling in this case
17:37:31 dansmith I'm describing what the usual nova error procedure is, regarding going to error state to signal to the user that their thing didn't happen
17:37:35 dansmith it's not great, it's just the convention
17:37:44 sean-k-mooney EugenMayer: creatign the glance image might require the instnace to be shutdown by the way
17:37:47 dansmith because if you do a backup, and the instance goes to active, you assume it worked, but it didn't
17:37:58 sean-k-mooney snapshots are not guareentee to be live
17:38:01 dansmith right
17:38:07 EugenMayer if nova is the task owner, which i understood is the case, it should design a propere state machine in case the task (which i understood is blocking via REST, so very fragile). Task could complete failed or succeeded. Task could never complete or even be deleted (on the glance side)
17:38:43 sean-k-mooney EugenMayer: there was a effort to do that at one point but this is also a distibuted system problem
17:38:49 dansmith EugenMayer: there's no task
17:38:49 EugenMayer understood, but i assume the sequence is: shutdown/sleep instance, create snapshot, start/resume instance, upload snapshot to glance .. (do task tracking)
17:39:42 sean-k-mooney EugenMayer:right yes but there may be clean up to be done in the compute node or stoage backend if the upload fails
17:40:20 EugenMayer no task means: it's blocking only. Understood there is no task_id or somewhat, just a blocking http-call. So as you both suggested, this blocking call needs to cleanup in all cases: 200,500 and also 408 and others.
17:40:22 sean-k-mooney such as deleting the file we created that was not uploaded
17:41:05 dansmith EugenMayer: we're saying exactly that.. we should, assuming we can
17:41:11 sean-k-mooney EugenMayer: yep and nova shoudl check the respocne code and start cleaning up if it failed
17:41:12 EugenMayer the i have seen the glance image task under image, which i was able to delete, but since the blocking request disconnected far ago, no cleanup happened on the nova side
17:41:28 dansmith EugenMayer: there are cases that are more complicated, such as with ceph where we might not be able to recover at all, depending on what happened, but in general we agree
17:41:41 EugenMayer agreed
17:42:16 sean-k-mooney well recovery in ceph might be squash/merge the ceph snapshot back into the previous volume for example
17:42:36 dansmith depends on the failure of course
17:42:54 sean-k-mooney where as for qcow we woudl mirror the file on disk then upload and if it faile delete the copy
17:43:45 sean-k-mooney EugenMayer: if you have logs and or a repoduce please file a bug and we can see if we can figure out why nova is not cleaning up as expected
17:52:07 opendevreview Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657
17:52:08 opendevreview Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to top-level modules https://review.opendev.org/c/openstack/nova/+/705658
17:52:08 opendevreview Stephen Finucane proposed openstack/nova master: trivial: Clean manager.Manager, service.Service signatures https://review.opendev.org/c/openstack/nova/+/764806
17:58:03 EugenMayer sean-k-mooney dansmith will do, thank you for your both time
18:22:04 admin1 hi all .. i am hitting this bug, https://bugs.launchpad.net/glance/+bug/1916482 , but don't have an idea on how to solve it .. i am using openstack-ansible and the latest tag 24.0.1
18:22:12 admin1 nova is local disk, glance is rbd
18:22:38 opendevreview Stephen Finucane proposed openstack/nova master: objects: Remove unnecessary type aliases, exceptions https://review.opendev.org/c/openstack/nova/+/738240
18:22:39 opendevreview Stephen Finucane proposed openstack/nova master: objects: Use imports instead of type aliases https://review.opendev.org/c/openstack/nova/+/738018
18:22:39 opendevreview Stephen Finucane proposed openstack/nova master: objects: Remove wrappers around ovo mixins https://review.opendev.org/c/openstack/nova/+/738019
18:22:40 opendevreview Stephen Finucane proposed openstack/nova master: WIP: add ovo-mypy-plugin to type hinting o.vos https://review.opendev.org/c/openstack/nova/+/758851
18:36:31 opendevreview Ghanshyam proposed openstack/nova stable/xena: DNM: testing centos8 py36 job https://review.opendev.org/c/openstack/nova/+/834765
18:38:10 opendevreview Ghanshyam proposed openstack/nova stable/wallaby: DNM: testing centos8 py36 job https://review.opendev.org/c/openstack/nova/+/834721
#openstack-nova - 2022-03-23
00:16:37 EugenMayer is it possible to somehow manually
00:16:52 EugenMayer (database manipulation) set the flavor id of an instance?
00:38:47 opendevreview anguoming proposed openstack/nova master: fix the bug of the log line has no request_id info at source host when live migration https://review.opendev.org/c/openstack/nova/+/834677
00:38:51 EugenMayer well i just created a new flavor of the same (old size) and then changed the flavorid in the nova_api database to match the expected old flavor id. Interestingly i could not find the relation table from flavor<->instance. it's not in nova::instances nor in nova_api:: anyway - where is this relation stored?
00:39:40 sean-k-mooney[m] its not we embed a copy of the flavor in the instance_extra table
00:40:17 sean-k-mooney[m] i belive we store teh orginal flavor name in the instnace in the api db
00:41:02 sean-k-mooney[m] but the only supported way to change the flaovor is via a resize
00:41:15 EugenMayer well you cannot do that if terraform bricked it
00:41:38 EugenMayer if a flav has been deleted while still the relation exists, which happened, you can neither resize nor do anything with this instance
00:41:39 sean-k-mooney[m] you should be able to do it from the api
00:41:47 EugenMayer no it wont work - 409
00:42:06 EugenMayer i tried it all via api before touching the database manually - for the obvious reasons
00:42:28 sean-k-mooney[m] you should be able to resize the instnace even if the flavor has been deleted
00:42:50 EugenMayer openstack server resize --flavor <newid> <serverid> will end up throwing a 409 if the current flavorid, which is referenced, does not exist
00:43:04 EugenMayer > you should be able to resize the instnace even if the flavor has been deleted
00:43:11 sean-k-mooney[m] hum that sound like a bug
00:43:13 EugenMayer this is not possible, i just did try exactly that.
00:44:59 sean-k-mooney[m] i dont knwo why we would need the old flavor to exist
00:45:32 EugenMayer thank you for the hint with instance_extra - that would be the better way to change the flavor there. I'am aware that the flavor must match the current instance specs if i do it on the db layer - all i want is fixing the actual relation so the API is operating again
00:46:26 EugenMayer it does not need it, it will just try to load the entity and resolve all it's relation and as a side effect an exception will be thrown, that the flavor entity cannot be loaded
00:47:03 EugenMayer and that exception most probably bubbles up and ends up canceling the request
00:47:28 sean-k-mooney[m] the request spec has referneces to the flavor too in the api db
00:48:42 EugenMayer i really see that the entire 'distributed system' and microservice architecture introduced a lot of issue here. We have a database nova_api defining ::flavor. It's PK is used as in FK in nova::instance_extra but without any constraint or anything else - since the databases 'are seperated' - which they are not at all.
00:49:19 sean-k-mooney[m] they are we store copies of the full flavor
00:49:35 sean-k-mooney[m] so the request spec has the serisalsed flavor embeded in it
00:49:44 sean-k-mooney[m] which is used for schduling
00:50:37 sean-k-mooney[m] nova is not a collection of microservice by the way
00:50:57 sean-k-mooney[m] its a single distbuted service with multiple components that work together
00:52:50 EugenMayer why did the databases have been designed this way? AFAICS there are relations from one entity in one database to a different database - this really is not how this should be designed right. so nova::instance_extra.flavor<->nova_api::flavor.flavorid
00:53:20 sean-k-mooney[m] that is not how it works
00:53:42 sean-k-mooney[m] when you boot an instance we make a copy of the flavor and store a copy in the instance table
00:53:55 EugenMayer in which instance table:
00:53:57 sean-k-mooney[m] flavor are imuntable and cant be change once created
00:54:05 sean-k-mooney[m] however flavor extra specs are mutable
00:54:06 EugenMayer nova::instances ?
00:54:28 sean-k-mooney[m] so we have to copy the flavor to make sure existing instnace are not change if you modify the extra specs
00:55:26 EugenMayer i guess you the then reference nova::instances.vcpus/memory_mb/disk_db and so on
00:56:20 sean-k-mooney[m] yes those are fixed and cannot be modified.
00:56:35 EugenMayer while i understand that flavors are immutable (and should be) and the actual fields are copied / flatted into nova::instances -there is still a relation from nova::instance_extra to nova_api:falvor
00:57:00 sean-k-mooney[m] yes and no
00:57:16 EugenMayer and if this relation is broken, which very well can happen since there is no constraint possible - you can no longer do anything with the instance. You can no longer apply any new flavor or resize it
00:57:49 EugenMayer so entire decoupling has been planned maybe, but it seems not have been implemented (yet)
00:57:52 sean-k-mooney[m] you shoudl be able to today
00:57:55 sean-k-mooney[m] if you cant its a bug
00:58:17 EugenMayer well then there is a bug in xena (not sure what today standard is, yoga or xena?)
00:58:53 sean-k-mooney[m] so you should be aware that flavors used to be defiend at the cell level with cells v1
00:59:31 sean-k-mooney[m] and when we moved to cells v2 we change the relation ships such that flavor were defiend gloablly in the api db

Earlier   Later