| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-02 | |||
| 10:26:24 | sean-k-mooney | melwitt: gibi: quick question about consumer types. will there be an api that allows me to list all allocation by consumer types and optionall filter that | |
| 10:27:55 | sean-k-mooney | for filtering i was thinking by host, project and maybe consumer/vm id | |
| 10:28:52 | sean-k-mooney | the reason im asking is the mailing list thread related to premptible instnaces. | |
| 10:29:35 | sean-k-mooney | one of the suggtions i had was to explore using a "premptible" consumer type for premtiable instances | |
| 10:30:29 | sean-k-mooney | initally if we had a query api it would allow and external service to esaisly determing what instance could be killed via a placment query | |
| 10:31:42 | sean-k-mooney | but i coudl imaging later extending plamcnet so that it coudl optionalling include the capsity used by preamtiable instance in its allocation candates set with a new micorverion and query arg | |
| 10:33:15 | sean-k-mooney | not that im going to work on that any time soon but i just taught that would enable and intersting desgin direction to explore | |
| 10:37:14 | jkulik | lyarwood, sean-k-mooney: wouldnt "having the api create the bdms" mean, that we need distributed locking between the apis, so we don't run into races while creating those? | |
| 10:38:07 | lyarwood | We'd need rely on the DB ensuring there's only ever one active BDM for a given instance UUID and volume ID combo | |
| 10:38:07 | sean-k-mooney | jkulik: we are creating them in the db so we can do a db level lock if we needed too | |
| 10:38:42 | jkulik | is multi-attach possible for the same VM? | |
| 10:38:50 | lyarwood | not for the same instance no | |
| 10:38:57 | sean-k-mooney | jkulik: within the same vm no | |
| 10:39:54 | sean-k-mooney | if we need a lock we can take a row level lock on the instance uuid or instance and volumne uuid pair | |
| 10:40:20 | lyarwood | tbh a constraint should be enough | |
| 10:40:44 | lyarwood | at least that's what was agreed on in the past when this came up | |
| 10:41:05 | lyarwood | but that was in the context of us creating duplicate BDMs on failure | |
| 10:41:15 | lyarwood | *failure to attach | |
| 10:41:20 | lyarwood | that hasn't been an issue for a while now | |
| 10:41:36 | sean-k-mooney | ya there is no index in the bdm that we need to worry about is there | |
| 10:41:44 | jkulik | lyarwood: "for a while" - which version do I need to upgrade to? :D | |
| 10:42:06 | lyarwood | jkulik: this was way back in Kilo I think and only reported by one customer downstream | |
| 10:42:12 | lyarwood | I've never seen it since | |
| 10:42:37 | lyarwood | -> lunch brb | |
| 10:42:43 | jkulik | hm ... I'm on rocky and need to investigate why there are like 10 bdm entries (pretty much empty) and one filled with connection_info in my DB for multiple VMs | |
| 10:43:02 | jkulik | same (volume_id, instance_uuid) | |
| 10:44:24 | sean-k-mooney | the uniqe constrati on rocky is just the bdm uuid | |
| 10:44:26 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/db/sqlalchemy/models.py#L587 | |
| 10:45:01 | sean-k-mooney | thats also the case for master | |
| 10:45:04 | sean-k-mooney | lyarwood: ^ | |
| 10:45:26 | stephenfin | bauzas: Friendly reminder that the 'instance_type' cleanup series is still waiting your attention, if you have some time https://review.opendev.org/q/topic:%2522compute_rpc_6.0%2522+status:open | |
| 10:45:35 | sean-k-mooney | so the uniqe constraint in the db i currently not enough to prevent duplicats | |
| 10:45:37 | lyarwood | jkulik: yeah the others are marked as soft deleted (deleted=id etc) | |
| 10:45:52 | lyarwood | sean-k-mooney: yup this is a new constraint I'm talking about | |
| 10:46:01 | sean-k-mooney | oh ok | |
| 10:46:01 | jkulik | no. they're not soft-deleted | |
| 10:46:14 | sean-k-mooney | well currently the could be not marked as soft deleted | |
| 10:46:35 | sean-k-mooney | you can have duplicate entries with the current schema | |
| 10:46:54 | lyarwood | right but that's the race I was saying we hadn't seen reported for a while | |
| 10:47:09 | sean-k-mooney | ah ok | |
| 10:47:31 | lyarwood | jkulik: what does `openstack server volume list $instance` show? | |
| 10:47:53 | lyarwood | jkulik: the same duplicates? | |
| 10:48:06 | sean-k-mooney | well backportbale way to fix this is to add a db trigger on insert to enforce the constraint then on master update the uniqe constratint | |
| 10:48:08 | jkulik | lyarwood: let me find such an instance again. I cleaned them up already for the customer to continue | |
| 10:49:30 | lyarwood | jkulik: ah if this is outside of the instance creation flow there have been known and now resolved issues with things like live migration rollbacks etc that could cause this | |
| 10:49:40 | lyarwood | and/or volume attach flows I should say | |
| 10:50:44 | jkulik | we're on VMware, so no live-migration via Nova for us in rocky afaik. might be volume-attach flows then | |
| 10:51:57 | jkulik | if they're fixed, I'll have a look through the commits. let's see what I can dig up. thank you | |
| 10:52:46 | stephenfin | lyarwood: Replied on https://review.opendev.org/c/openstack/nova/+/794006 I wasn't hitting the code path you're expecting because I wasn't booting from volume | |
| 10:53:38 | opendevreview | sean mooney proposed openstack/nova-specs master: Add no user token when get Cyborg client https://review.opendev.org/c/openstack/nova-specs/+/787178 | |
| 10:53:39 | stephenfin | that's evidently a far less common request than boot from volume, which explains why nobody has (based on reports, anyway) hit this before | |
| 10:54:51 | jkulik | lyarwood: I didn't even know "server volume list" was a thing. cool. it shows both bdm entries for the instance I found | |
| 10:56:03 | sean-k-mooney | stephenfin: boot from volumen is often an optimasation fo boot form exsiting volume | |
| 10:56:21 | lyarwood | jkulik: cool, and `openstack server event list $instance` shows just attaching the volume etc? | |
| 10:56:56 | sean-k-mooney | i would expect the latter to maybe be more common for heat or horizon users where you are less likely to typo the uuid due to the indirection | |
| 10:57:54 | sean-k-mooney | stephenfin: booting form an existing volume used to be common if you were creating the volum using an iso | |
| 10:57:56 | jkulik | lyarwood: hm ... hard to say. k8s instance with lot's of attachments going on. but it looks like in this case the volume is in state error | |
| 10:58:13 | sean-k-mooney | stephenfin: where boot form volume the normal way will not do what you want it too | |
| 10:58:20 | stephenfin | sean-k-mooney: tbc, I'm not doing boot from volume here. I'm booting from an image and attaching an additional volume | |
| 10:58:26 | stephenfin | so that's the unusual path, I think | |
| 10:58:37 | lyarwood | good old k8s, kk then this could be an issue with the vmware driver not cleaning up correctly on failure | |
| 10:58:41 | sean-k-mooney | not really i do that all the time | |
| 10:58:52 | stephenfin | i.e. 'server create --image IMAGE --block-device BDM ...' | |
| 10:59:00 | sean-k-mooney | that is proably the most common use of cinder | |
| 10:59:21 | stephenfin | clearly you're fastidious when it comes to your use of UUIDs so :) | |
| 10:59:38 | lyarwood | yeah that's a common use case, I wouldn't say the most common but it's up there | |
| 10:59:41 | sean-k-mooney | i never use the bdms direclly | |
| 10:59:56 | lyarwood | k8s etc boots and then attaches for example | |
| 10:59:59 | sean-k-mooney | i used to use --volume | |
| 11:00:11 | lyarwood | anyway stephenfin I think this was just missed tbh | |
| 11:00:18 | stephenfin | sean-k-mooney: --volume is for a boot device though | |
| 11:00:27 | lyarwood | I got slightly confused by some output you posted in here | |
| 11:00:29 | sean-k-mooney | no its not | |
| 11:00:29 | stephenfin | at least in OSC | |
| 11:00:35 | stephenfin | it sets boot_index to 0 | |
| 11:00:50 | lyarwood | yeah --volume is shorthand for bfv in osc | |
| 11:01:07 | sean-k-mooney | ok i normally use horizon for this but im pretty sure i used --volume for this too | |
| 11:01:23 | sean-k-mooney | that a pretty bad design choice then | |
| 11:01:30 | stephenfin | Yeah, I'm not a fan either | |
| 11:01:57 | stephenfin | it should be the equivalent of '--network' and '--port', which are shorthand for various permutations of '--nic' | |
| 11:01:59 | lyarwood | didn't you have a change up to move that to --boot-volume? | |
| 11:02:12 | stephenfin | not me, that I recall | |
| 11:02:20 | lyarwood | okay I thought someone did | |
| 11:02:38 | lyarwood | I'd be fine with that but I'm not sure how you could then reuse --volume without breaking people | |
| 11:02:47 | sean-k-mooney | --bfv was porposed at one point | |
| 11:03:37 | stephenfin | I guess you could consider ordering and use 'parsed_args.volumes[0]' is 'parsed_args.image_uuid' was unset? | |
| 11:03:39 | sean-k-mooney | lyarwood: given osc is ment to work the same acroos multiple clouds and this would not be a micorverison change | |
| 11:03:42 | stephenfin | s/is/if/ | |
| 11:03:51 | sean-k-mooney | we coudl only resue --volume after a major version bump | |
| 11:04:32 | stephenfin | not really, iirc using '--volume' and '--image' together currently is illegal | |
| 11:04:46 | stephenfin | so we'd be loosening restrictions, not adding new ones | |
| 11:05:37 | stephenfin | if it's not it probably should be, since I can't see how specifying an image_uuid and a BDM with a boot_index of 0 would be allowed by the server | |
| 11:05:39 | sean-k-mooney | you can use --image and --boot-from-volume <size> today that is the short hand for that | |
| 11:05:56 | stephenfin | yeah, that's not what I'm describing though and it's a different thing | |
| 11:05:58 | sean-k-mooney | but i tought we could use --image and --volume today to boot form image and attach data volumes | |
| 11:06:04 | stephenfin | that'll create a new volume iirc | |
| 11:06:12 | sean-k-mooney | can we add --data-volumn | |
| 11:06:25 | stephenfin | that's kind of ugly | |
| 11:07:14 | sean-k-mooney | well if we cant reclaim --volume then its kind of the best we can do | |
| 11:07:39 | sean-k-mooney | som of the osc optionts are kind of questionable | |