| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-09-04 | |||
| 18:30:43 | sean-k-mooney | that was randomly returnign a device busy sortly before/after onlining a core | |
| 18:30:53 | sean-k-mooney | as it was still goting though the state machine in the kernel | |
| 18:31:11 | sean-k-mooney | im not sure if proc would have the same issue | |
| 18:31:18 | dansmith | sean-k-mooney: right this code would not run on macos and so if it's naked in tests will break that compatibility we just achieved | |
| 18:31:22 | sean-k-mooney | since its not actully reading form hardware | |
| 18:31:43 | gouthamr | sean-k-mooney: on the self-healing at the host level vs giving the operator means.. we thought about this earlier.. draining a access needs host-local state (my_shared_fs_storage_ip/CONF.host + "is the share still hard mounted here").. operators could disable this periodic check though and let this only occur at startup (like other optional periodic tasks) | |
| 18:31:52 | sean-k-mooney | oh well yes that too | |
| 18:32:10 | dansmith | yeah making this not self-heal is bad, IMHO | |
| 18:32:21 | dansmith | and requiring cron running nova-manage on compute nodes is no bueno | |
| 18:32:50 | sean-k-mooney | im not agaisnt self healing i just suggested providing a command because we evnetull moved form healign the allcoation automaticly to a commnd | |
| 18:33:10 | sean-k-mooney | but ya i assume its like cider where we neded lcoal state to do that form brick | |
| 18:33:12 | dansmith | yeah but the command has to be run on each compute node, not centrally | |
| 18:33:13 | gouthamr | technically, on macos, you're only running unit tests? | |
| 18:33:28 | sean-k-mooney | dansmith: yes like with volumes | |
| 18:33:56 | sean-k-mooney | wich i also agree is kind of a pain so we can likely park that | |
| 18:34:07 | sean-k-mooney | so the prople is how to do that without melting the rpc bus | |
| 18:34:18 | sean-k-mooney | *problem | |
| 18:34:57 | gouthamr | (i mocked the /proc/mounts calls .. i run unit tests locally on mac, and yes, this would _have_ to be mocked regardless) | |
| 18:36:01 | sean-k-mooney | it sound like a direct remotable obejct method on InstanceList that retuns the list of instnace that have attachment on a given host with the atachmetn info is going to be needed | |
| 18:36:12 | gouthamr | (ese: https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/tests/unit/compute/test_compute_mgr.py#4435) | |
| 18:36:30 | sean-k-mooney | so one rpc that give you only the instnace for a given host that have attachmets with the attachment info | |
| 18:36:53 | sean-k-mooney | dansmith: was the concer only with instnace the were moved sepcificly? | |
| 18:38:57 | sean-k-mooney | doing that filtering on the python side seam like a bad time but we shoudl be able to do it semi effeictly in sql | |
| 18:39:38 | dansmith | gouthamr: mm, nope functional should be working (I'm running now to see if something has regressed) | |
| 18:40:09 | sean-k-mooney | we would need to join on the migration table compute nodes tabel and isntance table and isntance sares table but we shoudl be abel to say give me all instnace that migrated to/from this host that had share attachmetns | |
| 18:40:16 | dansmith | sean-k-mooney: and yeah sounds like maybe a single query method that returns "any share mapping we might care about" would be the best option.. we have a ShareMappingList already which is the right place to put it I thinjk | |
| 18:40:17 | gouthamr | ack, i ran functional too on a mac.. /me is running it again after collapsing the queries | |
| 18:40:50 | sean-k-mooney | ya if its the mappitn we care about that sound better to me as well | |
| 18:41:01 | sean-k-mooney | i was not sure if we wanted the mappitns or the instnace or both | |
| 18:41:12 | dansmith | sean-k-mooney: I'm going to commit comments on the concerns.. would be nice to have your weight applied | |
| 18:41:16 | dansmith | sean-k-mooney: mappings AFAIK | |
| 18:41:26 | gouthamr | yes, mappings | |
| 18:42:13 | sean-k-mooney | sure but as the questions i have been asking imply i have not been folloing the overall work closely | |
| 18:42:18 | melwitt | yeah so it would be get all share mappings for a list of instance uuids rather than doing one by one right | |
| 18:42:26 | sean-k-mooney | but i agree the scalaiblity of the current approch is concerning | |
| 18:42:29 | gouthamr | yeah.. the data model was forcing me with some of these calls in the code.. I'm working on your suggestions to stitch (InstanceList + MigrationList + ShareMappingList) and get a list to parse... one remotable RPC that does all this would help cleanly for sure. but, can i do that in Indri? or would you prefer to have it done now? | |
| 18:45:36 | melwitt | adding a ShareMappingList.get_by_instance_uuids() would be simple enough. do we need more than that? | |
| 18:45:52 | dansmith | gouthamr: what you just described is what I just said we should do in my comment just now | |
| 18:46:15 | sean-k-mooney | or ShareMappingList.get_by_migration_host ? | |
| 18:46:46 | dansmith | gouthamr: I don't think we can/should merge this the way it is, but it would suck to have to punt it all to indri as well.. since you'll have to add RPC changes it won't be backportable either | |
| 18:47:00 | dansmith | melwitt: instance uuids is not enough and it's not even the likely larger list | |
| 18:47:02 | gouthamr | yeah no.. i wasn't looking to backport this stuff | |
| 18:47:48 | gouthamr | its super important for me though to make forward progress on the cold migration :( it makes all of this partially usable before we fix up the hard block on other instance actions that you folks pointed out at the PTG | |
| 18:48:20 | gouthamr | the backport was just going to be for: https://review.opendev.org/c/openstack/nova/+/1001900/ | |
| 18:48:26 | dansmith | yup | |
| 18:48:31 | gouthamr | s/backport/backport request :) | |
| 18:49:41 | gouthamr | ty for this discussion and the comments! | |
| 18:53:15 | sean-k-mooney | gouthamr: i have not looked at https://review.opendev.org/c/openstack/nova/+/1001900 in detail but just looking at the tile im not sure that is backportable | |
| 18:53:56 | gouthamr | sean-k-mooney: oh, read on :P yes, we're moving auth, but it doesn't break anything running until you choose | |
| 18:54:11 | sean-k-mooney | it might be but we have to supprot n-2 compatiblity | |
| 18:54:15 | gouthamr | sean-k-mooney: this will be supplemented with an OSSN/operator guidance on how the move will occur | |
| 18:55:16 | sean-k-mooney | so we need atuth to work with a patched souce node and unpatch dest | |
| 18:55:20 | sean-k-mooney | as an example | |
| 18:55:37 | sean-k-mooney | its too late on a friday for me to really ingest the details | |
| 18:55:56 | gouthamr | sean-k-mooney: it will; on the unpatched destination, we'll use a "shared credential" until that's patched some day | |
| 18:56:32 | gouthamr | sean-k-mooney: yes :) ty for the brainstorm! have a great weekend | |
| 18:57:44 | dansmith | sean-k-mooney: it's really not ideal, I've already raised the concerns | |
| 18:58:13 | sean-k-mooney | ack, im not really sure that shareing a cephx id is a problme | |
| 18:58:17 | dansmith | sean-k-mooney: the virtiofs implementation in the tree is sort of "so leaky that it doesn't break upgrades" which is not good but also means that we _can't_ do automatic cleanup there anyway | |
| 18:58:22 | sean-k-mooney | but i woudl have to read the bug to knwo why it woudl be | |
| 18:58:33 | dansmith | sean-k-mooney: it's a hardening thing | |
| 18:59:01 | sean-k-mooney | sure but for nova rbd driver we mostly use the same cephx key | |
| 18:59:07 | sean-k-mooney | technially it can vary per host | |
| 18:59:14 | sean-k-mooney | but i dont really knwo of any installer that does that | |
| 18:59:37 | sean-k-mooney | we just use a single nova key on all compute nodes | |
| 19:00:32 | sean-k-mooney | so using shared ceph keyrings between host is not generally a sefutiy issue but i dont really have an object to hardening | |
| 19:01:06 | dansmith | I'm not sure our bad behavior is a justification to do it badly.. but those ceph shares are less user-visible than these I think | |
| 19:01:20 | dansmith | and the current implementation sort of can't even track which hosts should and should not have access to a share as a result | |
| 19:01:44 | sean-k-mooney | ack as i said i have no real issue with hardenign | |
| 19:01:58 | sean-k-mooney | jsut notign that it not the same as nova provisend rbd voluems | |
| 19:02:11 | sean-k-mooney | i think for cidner we are given a cephx key in teh conenciton info to ues | |
| 19:02:16 | sean-k-mooney | possibel per voluem? | |
| 19:02:50 | sean-k-mooney | and as i said ther is nothign prventign you minting a per host key for images_type=rbd today | |
| 19:03:21 | sean-k-mooney | but ya as you said if you have escaped to the host your already have other compomise vector to worry about | |
| 19:07:07 | sean-k-mooney | for the peroupus of share attachment the client of the sheare was desiged to be nova rather then the instnace | |
| 19:07:47 | sean-k-mooney | the same way as nova is the sincel client for ceph voleusm for images_type=rbd | |
| 19:07:53 | dansmith | that's how the code is designed now, but I think that was probably wrong | |
| 19:08:26 | sean-k-mooney | it depend on your pesertive of who own the expore the end user or nova | |
| 19:08:35 | sean-k-mooney | if its nova which is howe we desgiend it its correct | |
| 19:08:40 | dansmith | treating all of nova as a single unit is a bad idea, whether it be shares, volumes, access to the MQ or DB, etc.. in 2010 I think that was a lot more normal but today it seems verging on irresponsible to me | |
| 19:08:48 | gouthamr | ++ | |
| 19:08:58 | sean-k-mooney | if its the end user then the cephex key shoudl be in barbican liek cinder encyption keys | |
| 19:09:49 | sean-k-mooney | right as i said im oke with hardenign but that feels mroe like a security feature then soemthign we would activly backport | |
| 19:10:02 | gouthamr | cephfs != rbd, for RBD, cephx is one of several layers. For a CephFS share, the cephx access rule is the access-control layer.. | |
| 19:10:09 | dansmith | it is and you could surely argue that it's not backport-worthy IMHO | |
| 19:10:35 | gouthamr | even there though, we could make a better choice than having a single shared credential, imo.. | |
| 19:11:01 | sean-k-mooney | gouthamr: we dont require it to be shared in nova today taht up to the installer | |
| 19:11:02 | melwitt | so are we thinking a separate patch for the ShareMappingList.get_share_ids_for_host() to go in front of the reconcile patch? | |
| 19:11:27 | gouthamr | melwitt: i'll fold it in.. | |
| 19:11:37 | melwitt | to try and minimize churn on the reconcile? ok | |
| 19:11:48 | gouthamr | sean-k-mooney: ack, yes, but in this case, it's hardcoded so worse :( | |
| 19:12:12 | gouthamr | melwitt: it was the last patch in the series, so i can fix it up in isolation :) | |
| 19:13:31 | gouthamr | dansmith: i'm exploring psutil.disk_partitions.. i see its being used by quobyte: https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/volume/quobyte.py#L112 | |
| 19:14:04 | dansmith | gouthamr: aha, nice | |
| 19:14:06 | gouthamr | dansmith: it solves my problem of not hemorrhaging on an inaccessible hard mount | |
| 19:16:13 | dansmith | so realistically here.. unfortunately monday is a holiday for many of us | |
| 19:16:29 | dansmith | I wish Uggla was around but I assume merging this on tuesday is pretty dang late | |
| 19:16:48 | dansmith | I almost wish melwitt had been less of a responsible person and sent the bottom patches to the gate already :) | |
| 19:17:54 | sean-k-mooney | so realsticlly , we cut rc1 without this next week, we can still merge this on master it just will be for next cycel if we reapprove the spec | |
| 19:18:14 | sean-k-mooney | and we can repprose that already | |