| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-31 | |||
| 00:55:49 | tonyb | Any advice on how to reconcile the list with the instances that are still alive? | |
| 06:03:18 | tonyb | Picking one instance as an example, I see it in nova_api.instance_mapping, looking in the nove_cell0.instances it shows as deleted. | |
| 11:30:53 | sean-k-mooney[m] | @tonybso cell0 is only used for instances that fail to boot to any host but pass the intal api checks for is this request valid | |
| 11:31:23 | sean-k-mooney[m] | cell0 should never have any compute nodes in it | |
| 11:38:56 | tonyb | I did more digging and found a number (but not all) of the instances showing up in 'server list' have data in the build_requests table. | |
| 11:39:55 | tonyb | I *suspect* I can delete them from the build_requests table and they | |
| 11:40:12 | tonyb | 'll "go away" | |
| 11:42:06 | tonyb | I also expect that will take care of most of the problems and then I can work out what the next stage is | |
| 11:51:09 | sean-k-mooney[m] | that could happen in older openstack release but i belive we fixed that at some point | |
| 11:52:22 | sean-k-mooney[m] | has anyone worked with the SovereignCloudStack folks on there standards openstack clouds | |
| 11:52:26 | sean-k-mooney[m] | https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0100-v3-flavor-naming.md | |
| 11:53:02 | sean-k-mooney[m] | that is interestign and i agree with most of it but it also impleies some functionality that nova does not supprot nativly | |
| 11:53:13 | sean-k-mooney[m] | at lest not without security risk | |
| 11:54:04 | sean-k-mooney[m] | there disk section https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0100-v3-flavor-naming.md#baseline-2 | |
| 11:54:40 | sean-k-mooney[m] | the mention if the flaovr name does not have a disk capsity that menas the cloud will provide at least image.min_disk space | |
| 11:55:23 | sean-k-mooney[m] | that either need the operator to cofnig root_gb=0 and allow non bfv instance with 0 root disk | |
| 11:55:31 | sean-k-mooney[m] | that was considred a cve in the past | |
| 11:55:49 | sean-k-mooney[m] | or they need to have downstrema changes to supprot that | |
| 11:56:39 | sean-k-mooney[m] | i feel its in approate to propsoe a standard behavior that is not supproted upstream or that depends on a security vulnerability | |
| 11:57:01 | sean-k-mooney[m] | the also have | |
| 11:57:02 | sean-k-mooney[m] | Multi-provisioned Disk | |
| 11:57:02 | sean-k-mooney[m] | The disk size can be prefixed with Mx prefix, where M is an integer specifying that the disk is provisioned M times. Multiple disks provided this way should be independent storage media, so users can expect some level of parallelism and independence. | |
| 11:57:24 | sean-k-mooney[m] | nova only support multipel disks for the same isntance if you use both root_gb adn ephmeral_gb | |
| 11:57:36 | sean-k-mooney[m] | and nova does not supprot puttign them on diffent storage media | |
| 11:57:52 | sean-k-mooney[m] | so again that would require downstream only modifications to nova | |
| 11:57:54 | tonyb | sean-k-mooney[m]: This is victoria | |
| 11:58:14 | sean-k-mooney[m] | @tonyb ya i dont think it was fixed then | |
| 11:58:35 | sean-k-mooney[m] | there was an edge case where if an error happened durign delete sometimes the build request didnt get cleaned up properly | |
| 11:59:22 | tonyb | Okay, rabbitmq died on this cloud (I don't know why) so I can see events going missing/. | |
| 12:00:11 | sean-k-mooney[m] | did db access fail | |
| 12:00:21 | sean-k-mooney[m] | i dont think it happend because of rabbit but perhaps | |
| 12:00:30 | tonyb | "safe" is the wrong word but do you think it'll be "safe" to just delete those entries from build_requests | |
| 12:01:02 | tonyb | I don't think the DB access failed | |
| 12:01:13 | sean-k-mooney[m] | if the instance are already in cell one or marked as deleted then ya it should be “safe” | |
| 12:03:03 | tonyb | sean-k-mooney[m]: Thanks I'll try a couple tomorrow and see what happens | |
| 12:03:57 | tonyb | It'd be nice to clean this up as it's one of our nodepool providers | |
| 12:15:07 | auniyal | Hi gibi, can you please have a look on this some time https://review.opendev.org/q/topic:%22bp/2048184%22 | |
| 12:34:05 | auniyal | thanks sean-k-mooney[m]++ | |
| 12:35:01 | sean-k-mooney | no worreies. im mostly done with upstream code review for today bu ti also reviewd your other patch this morning | |
| 12:35:05 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/857339 | |
| 12:36:51 | auniyal | yeah, I'll udpate this as well | |
| 12:38:53 | auniyal | if you still have some time, can you please see this as well - https://review.opendev.org/q/topic:%22refactor-vf-profile-update%22 | |
| 12:39:07 | auniyal | it will help this - https://review.opendev.org/c/openstack/nova/+/899253/4 | |
| 12:43:01 | frickler | sean-k-mooney: I'll forward your comments regarding SCS, seems noone is around in this channel | |
| 12:46:05 | sean-k-mooney | frickler: cool. i dont dislike the idea of having standard around this | |
| 12:46:31 | sean-k-mooney | but i would hope those stanard would start form a point of what is secure and supproted in vanialla openstack | |
| 12:46:47 | sean-k-mooney | frickler: the multi disk capablity could work if your using ironic | |
| 12:47:14 | sean-k-mooney | i.e. you could have seperate storage media for strage provided by hte flavor in that case | |
| 12:47:31 | sean-k-mooney | but i dont think its possibel for any other virt dirver and it defintly is not for libvirt | |
| 13:26:52 | Uggla | gibi, bauzas question for unmounting a share. There is no polling in that case but as we are calling manila, I guess it should be asynchronous too. Right ? | |
| 13:27:29 | bauzas | when would you unmount the share ? | |
| 13:27:42 | Uggla | bauzas, it depends. :) | |
| 13:27:51 | bauzas | of the snow level ? | |
| 13:27:53 | bauzas | :) | |
| 13:28:06 | Uggla | bauzas, if it is not used by another VM yes. | |
| 13:28:23 | bauzas | so when deleting the instance ? | |
| 13:30:27 | Uggla | bauzas, hum not sure I get your point. But if you delete a VM with a share it goes to the same code and umount the share if it is not used elsewhere. | |
| 13:40:45 | gibi | Uggla: yeah, as we are calling other APIs I would make the detach async as well. But as you pointed out that will be faster as no polling there. | |
| 13:40:50 | bauzas | so yeah, it should be an async call | |
| 13:41:00 | bauzas | by design, we never ever stop a deletion | |
| 13:41:25 | bauzas | we first accept the deletion request and only after we try to delete the resource | |
| 13:42:43 | gibi | bauzas: having the unmount at detach means VM deletion does not need to talk to manila | |
| 13:42:57 | gibi | scratch that | |
| 13:43:16 | gibi | if we delete a VM with an attached share then the umount need to happen during VM deletion | |
| 13:43:37 | gibi | but VM delet is already async so I don't see an issue there | |
| 13:43:45 | Uggla | gibi, ++ | |
| 13:47:09 | bauzas | gibi: you're right on the fact that the delete is already async | |
| 13:47:46 | bauzas | when thinking about that, I also wonder why we need to unmount the share when deleting the resource in the compute | |
| 13:47:50 | Uggla | gibi, I would say the polling is more to ensure manila has applied the policy before mounting than really wait for manila. I set a timeout here mainly to avoir an infinite loop if manila is not working. | |
| 13:48:07 | bauzas | it could be a periodic | |
| 13:48:24 | bauzas | (if we have correct statuses) | |
| 13:49:39 | Uggla | bauzas, yep but it is better to clean the room yourself and not your mom (periodic). :) | |
| 13:49:51 | gibi | I see periodics as fallbacks when something bad happens during normal operators. | |
| 13:50:03 | bauzas | my only concern is the polling | |
| 13:50:13 | bauzas | if we don't need to call manila, sure | |
| 13:50:28 | gibi | we need to call manial to unlock the share | |
| 13:50:40 | bauzas | hence my concern | |
| 13:50:43 | gibi | but during delete we don't need to poll manila | |
| 13:50:49 | gibi | as unlock is sync | |
| 13:50:51 | gibi | I think | |
| 13:51:02 | bauzas | there are two stages | |
| 13:51:07 | bauzas | 1/ user asks to delete the instance | |
| 13:51:15 | bauzas | 2/ compute eventually deletes it | |
| 13:51:29 | bauzas | surely, for 1/, it's async | |
| 13:51:50 | bauzas | the compute could be not running | |
| 13:51:51 | bauzas | but for 2/ | |
| 13:52:04 | bauzas | once the compute gets the fact that it needs to delete a guest, then it runs | |
| 13:52:15 | bauzas | and that's my concern | |
| 13:52:53 | bauzas | I don't want this compute run to be waiting to manila to unlock the share | |
| 13:53:20 | gibi | but we do the same with port unbinding and probably for unreserving the volume in glance | |
| 13:53:23 | gibi | in cinder | |
| 13:54:08 | bauzas | hmm ok | |
| 13:54:37 | gibi | also we tend to say that nova compute works sort of with all disable periodics | |
| 13:56:08 | bauzas | wdym ? | |
| 13:57:06 | sean-k-mooney | im in a meeting right now but in genreal we shoudl be able to run nova without perodics enabeld modulo bugs | |
| 13:57:14 | gibi | :) | |
| 13:57:20 | sean-k-mooney | so fallback to perodic sure | |
| 13:57:21 | gibi | yeah I heard this from Sean | |
| 13:57:40 | gibi | bauzas: we do the volume detach in cinder API during terminate instance here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3284 | |
| 13:58:24 | sean-k-mooney | the only "features" that need perodics to fucntion are things like local_delete | |