| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-22 | |||
| 17:22:58 | dansmith | everything is nova->glance | |
| 17:23:05 | sean-k-mooney | i belive this is a blocking call to do the upload to glace | |
| 17:23:15 | sean-k-mooney | if its async then either nova would poll | |
| 17:23:22 | sean-k-mooney | or we woudl get an external event form glance | |
| 17:23:26 | dansmith | so depending on the failure, nova should clean up whatever it can.. an upload to glance for sure should be recoverable on our end, so that's likely it's own bug if we're missing something | |
| 17:23:28 | sean-k-mooney | but i think image upload if blocking | |
| 17:23:35 | dansmith | sean-k-mooney: none of that with glance | |
| 17:24:00 | sean-k-mooney | right we dont do polling or external event right | |
| 17:24:07 | sean-k-mooney | we just do two blocking calls | |
| 17:24:09 | EugenMayer | if it is a blocking task, well the blocking should cleanup - which it seem to not do | |
| 17:24:15 | sean-k-mooney | one for creating the image and the second for the data upload | |
| 17:24:34 | dansmith | EugenMayer: if you can repro the problem that's definitely a bug candidate | |
| 17:24:42 | sean-k-mooney | EugenMayer: yes it should clean up if we get an error form glance | |
| 17:24:56 | dansmith | there are some situations where it might not make sense to clean up, but I would think a glance thing would always be something we can handle | |
| 17:25:04 | EugenMayer | dansmith i can reproduce this the 4th time. If you tell me what to gather, i will grab the logs you need the 5th time - which will happen | |
| 17:25:13 | dansmith | EugenMayer: logs | |
| 17:25:18 | EugenMayer | which logs to get? | |
| 17:25:26 | dansmith | all of them? :) | |
| 17:25:26 | sean-k-mooney | dansmith: i would expect the vm to go back to active or error if we dont clean up right | |
| 17:25:32 | dansmith | nova-compute, nova-api at least | |
| 17:25:36 | dansmith | sean-k-mooney: error, yeah | |
| 17:25:56 | EugenMayer | vm is in active state, power is on, task_state is image_backup | |
| 17:26:07 | dansmith | that said, reset_state resets task_state so that should be the way to get out here | |
| 17:26:23 | sean-k-mooney | you can reset state to active | |
| 17:26:39 | EugenMayer | reset-state --active + reboot seems to recover just right. Also viewing the console works (which is one of the problems with a partial state recovery) | |
| 17:26:40 | dansmith | EugenMayer: we're saying that what we would expect is vm_state=ERROR,task_state=None | |
| 17:26:44 | sean-k-mooney | rather then error and potentaly just trigger the backup/snapthot again | |
| 17:26:55 | EugenMayer | dansmith that never happened yet | |
| 17:27:07 | dansmith | EugenMayer: I know, I'm saying that's what we expect nova should be doing | |
| 17:27:17 | sean-k-mooney | EugenMayer: do you know why the glance operation is failing. | |
| 17:28:02 | sean-k-mooney | dansmith: i could see an argument to be made that we woudl have vm_state=Active task_state=None but the snapshot action was marked as error in the server event log | |
| 17:28:20 | sean-k-mooney | if the vm was indeed still runing proberly depending on how it failed | |
| 17:28:42 | EugenMayer | there is so much one can break right now. e.g. a other topic is using terraform and rescale a flavor. In 2 of 5 cases the following happens (i cannot tell you exactly). The old flavor is delete (too early), the new one is created, then the instance is fetched, this fails since the flavor_id of the old flavor is still set and cannot be found. TF | |
| 17:28:42 | EugenMayer | cancles and that's it | |
| 17:28:43 | dansmith | sean-k-mooney: the problem is one of signaling, which is why we (originally as designed) went to error,None for everything and then you do a start (which does nothing) to reset back to active as sort of "ack" | |
| 17:29:07 | EugenMayer | stuck again - stuck that one now needs to shelve the instance and restore it from glance using the 'new flavor' | |
| 17:30:01 | EugenMayer | i did not yet check the tf openstack provider implementation to see what they have implemented and how that is a timing issue in the first place (since it does not happen every timee) .. but if i look at the openstack rest api / nova api .. swapping flavors is not designed at all. | |
| 17:30:19 | sean-k-mooney | EugenMayer: well flavor are intenede to be imuatble so you idealy woudl not delete them until all instance using them are resized | |
| 17:30:28 | sean-k-mooney | we do cache the flavor | |
| 17:30:30 | sean-k-mooney | in the insntace | |
| 17:30:42 | dansmith | EugenMayer: are you describing two issues or one? if the former, then let's not complicate diagnosing this one | |
| 17:30:47 | sean-k-mooney | but really you shoudl try to avoid removing flavor or image that are in use | |
| 17:31:08 | EugenMayer | well i cannot tell why tf openstack providere deletes the flav too early or whatever happens in detail (i did not check the sequence in the code yet) | |
| 17:31:23 | sean-k-mooney | EugenMayer: it should not delete it at all | |
| 17:31:31 | EugenMayer | dansmith sorry, my bad. second issue (the latter one with the flav) | |
| 17:31:44 | sean-k-mooney | it sould like they are implementing the hacky workflow that horizon use to have | |
| 17:31:48 | dansmith | EugenMayer: yeah, not helping :) | |
| 17:32:00 | EugenMayer | dansmith sorry. my bad. | |
| 17:32:11 | sean-k-mooney | where they allowed you to update a flavor by deletign and recating it but ya lets not talk about that issue now | |
| 17:35:16 | EugenMayer | well if you ask me to the state error - one should not mark the instance as 'error' if a image_backup task failed - there is no reason for that. Creating a glance image does not required the instance to shutdown or similar, this said, i assume both task (the instance running) and the creation of the image can work in parallel and are independent | |
| 17:35:54 | dansmith | EugenMayer: going to error state is just the nova convention (in most places) | |
| 17:36:00 | EugenMayer | so this said, if the image_backup task is: failed, the task_id does no longer exists or whatever, nova should not block 'restarting the instance' | |
| 17:36:11 | dansmith | and if the issue wasn't critical, then a start operation will clear the error state without requiring a reboot of the actual instance | |
| 17:36:16 | EugenMayer | dansmith well it is the 'better safe then sorry convention i guess' | |
| 17:37:09 | dansmith | EugenMayer: we're agreeing with you that we do not expect that this is something that should be so jammed up and that there's probably some missing error handling in this case | |
| 17:37:31 | dansmith | I'm describing what the usual nova error procedure is, regarding going to error state to signal to the user that their thing didn't happen | |
| 17:37:35 | dansmith | it's not great, it's just the convention | |
| 17:37:44 | sean-k-mooney | EugenMayer: creatign the glance image might require the instnace to be shutdown by the way | |
| 17:37:47 | dansmith | because if you do a backup, and the instance goes to active, you assume it worked, but it didn't | |
| 17:37:58 | sean-k-mooney | snapshots are not guareentee to be live | |
| 17:38:01 | dansmith | right | |
| 17:38:07 | EugenMayer | if nova is the task owner, which i understood is the case, it should design a propere state machine in case the task (which i understood is blocking via REST, so very fragile). Task could complete failed or succeeded. Task could never complete or even be deleted (on the glance side) | |
| 17:38:43 | sean-k-mooney | EugenMayer: there was a effort to do that at one point but this is also a distibuted system problem | |
| 17:38:49 | dansmith | EugenMayer: there's no task | |
| 17:38:49 | EugenMayer | understood, but i assume the sequence is: shutdown/sleep instance, create snapshot, start/resume instance, upload snapshot to glance .. (do task tracking) | |
| 17:39:42 | sean-k-mooney | EugenMayer:right yes but there may be clean up to be done in the compute node or stoage backend if the upload fails | |
| 17:40:20 | EugenMayer | no task means: it's blocking only. Understood there is no task_id or somewhat, just a blocking http-call. So as you both suggested, this blocking call needs to cleanup in all cases: 200,500 and also 408 and others. | |
| 17:40:22 | sean-k-mooney | such as deleting the file we created that was not uploaded | |
| 17:41:05 | dansmith | EugenMayer: we're saying exactly that.. we should, assuming we can | |
| 17:41:11 | sean-k-mooney | EugenMayer: yep and nova shoudl check the respocne code and start cleaning up if it failed | |
| 17:41:12 | EugenMayer | the i have seen the glance image task under image, which i was able to delete, but since the blocking request disconnected far ago, no cleanup happened on the nova side | |
| 17:41:28 | dansmith | EugenMayer: there are cases that are more complicated, such as with ceph where we might not be able to recover at all, depending on what happened, but in general we agree | |
| 17:41:41 | EugenMayer | agreed | |
| 17:42:16 | sean-k-mooney | well recovery in ceph might be squash/merge the ceph snapshot back into the previous volume for example | |
| 17:42:36 | dansmith | depends on the failure of course | |
| 17:42:54 | sean-k-mooney | where as for qcow we woudl mirror the file on disk then upload and if it faile delete the copy | |
| 17:43:45 | sean-k-mooney | EugenMayer: if you have logs and or a repoduce please file a bug and we can see if we can figure out why nova is not cleaning up as expected | |
| 17:52:07 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/c/openstack/nova/+/705657 | |
| 17:52:08 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 17:52:08 | opendevreview | Stephen Finucane proposed openstack/nova master: trivial: Clean manager.Manager, service.Service signatures https://review.opendev.org/c/openstack/nova/+/764806 | |
| 17:58:03 | EugenMayer | sean-k-mooney dansmith will do, thank you for your both time | |
| 18:22:04 | admin1 | hi all .. i am hitting this bug, https://bugs.launchpad.net/glance/+bug/1916482 , but don't have an idea on how to solve it .. i am using openstack-ansible and the latest tag 24.0.1 | |
| 18:22:12 | admin1 | nova is local disk, glance is rbd | |
| 18:22:38 | opendevreview | Stephen Finucane proposed openstack/nova master: objects: Remove unnecessary type aliases, exceptions https://review.opendev.org/c/openstack/nova/+/738240 | |
| 18:22:39 | opendevreview | Stephen Finucane proposed openstack/nova master: objects: Use imports instead of type aliases https://review.opendev.org/c/openstack/nova/+/738018 | |
| 18:22:39 | opendevreview | Stephen Finucane proposed openstack/nova master: objects: Remove wrappers around ovo mixins https://review.opendev.org/c/openstack/nova/+/738019 | |
| 18:22:40 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: add ovo-mypy-plugin to type hinting o.vos https://review.opendev.org/c/openstack/nova/+/758851 | |
| 18:36:31 | opendevreview | Ghanshyam proposed openstack/nova stable/xena: DNM: testing centos8 py36 job https://review.opendev.org/c/openstack/nova/+/834765 | |
| 18:38:10 | opendevreview | Ghanshyam proposed openstack/nova stable/wallaby: DNM: testing centos8 py36 job https://review.opendev.org/c/openstack/nova/+/834721 | |
| #openstack-nova - 2022-03-23 | |||
| 00:16:37 | EugenMayer | is it possible to somehow manually | |
| 00:16:52 | EugenMayer | (database manipulation) set the flavor id of an instance? | |
| 00:38:47 | opendevreview | anguoming proposed openstack/nova master: fix the bug of the log line has no request_id info at source host when live migration https://review.opendev.org/c/openstack/nova/+/834677 | |
| 00:38:51 | EugenMayer | well i just created a new flavor of the same (old size) and then changed the flavorid in the nova_api database to match the expected old flavor id. Interestingly i could not find the relation table from flavor<->instance. it's not in nova::instances nor in nova_api:: anyway - where is this relation stored? | |
| 00:39:40 | sean-k-mooney[m] | its not we embed a copy of the flavor in the instance_extra table | |
| 00:40:17 | sean-k-mooney[m] | i belive we store teh orginal flavor name in the instnace in the api db | |
| 00:41:02 | sean-k-mooney[m] | but the only supported way to change the flaovor is via a resize | |
| 00:41:15 | EugenMayer | well you cannot do that if terraform bricked it | |
| 00:41:38 | EugenMayer | if a flav has been deleted while still the relation exists, which happened, you can neither resize nor do anything with this instance | |
| 00:41:39 | sean-k-mooney[m] | you should be able to do it from the api | |
| 00:41:47 | EugenMayer | no it wont work - 409 | |