| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-02 | |||
| 07:25:32 | yonglihe | * add cyborg ARQ clean up for all sorts of failure | |
| 07:25:32 | yonglihe | * put the device_profile into reqeust_net(API stage), then we elimate one neutron call. | |
| 07:25:32 | yonglihe | previous change set1: | |
| 07:25:33 | yonglihe | * reject create smatnic port attached server while migration in progress | |
| 07:25:33 | yonglihe | * bump compute service version number | |
| 07:26:46 | alex_xu | yonglihe: planning to review next around | |
| 07:38:45 | nightmare_unreal | is there a way to check hypervisor kernel version through openstack | |
| 08:13:45 | bauzas | nightmare_unreal: please ask your question on the new OFTC channel | |
| 08:14:12 | bauzas | we moved the community IRC channels to OFTC last weekend | |
| 08:14:14 | nightmare_unreal | can you poing to the name what's the #tag of that channel ? | |
| 08:14:22 | bauzas | same than here | |
| 08:14:26 | bauzas | s/freenode/oftc | |
| 08:14:32 | nightmare_unreal | thanks :) | |
| 08:14:36 | nightmare_unreal | s/freenode/oftc | |
| 08:15:03 | bauzas | you'll find the exactly same rooms in the OFTC network but you'll need to register your nick again | |
| 08:15:59 | nightmare_unreal | I see , we are moving away from freenode | |
| 08:18:05 | bauzas | yup | |
| 08:18:27 | bauzas | I stick here for redirecting folks that haven't seen it yet | |
| 08:18:41 | bauzas | EOF. | |
| 09:17:41 | opendevmeet | Launchpad bug 1930406 in OpenStack Compute (nova) "parallel volume-attachment requests might starve out nova-api for others" [Undecided,New] | |
| 09:17:41 | lyarwood | melwitt: https://bugs.launchpad.net/nova/+bug/1930406 - btw when you're around later can we chat about this and any workarounds you're aware of? | |
| 09:18:21 | lyarwood | but tbh I've wanted to drop the offending RPC call for a while and this might be the report that finally pushes me to remove it | |
| 09:18:31 | lyarwood | behind a microversion | |
| 09:18:49 | lyarwood | as it changes what we return to the caller | |
| 10:09:07 | sean-k-mooney | lyarwood is it just doing the rpc for the device name which we dont even garentee inside the guest | |
| 10:09:36 | lyarwood | yeah we also create the bdm during the same call | |
| 10:09:48 | lyarwood | via the conductor so it takes a little longer | |
| 10:10:48 | sean-k-mooney | should we make this an async api and just make it a cast | |
| 10:12:33 | sean-k-mooney | we have been talking about rpc version compatiablity downstream for a while on an off | |
| 10:12:45 | lyarwood | we need the BDM creation to be sync to avoid races but we can do that on the api | |
| 10:12:53 | lyarwood | and then move this device assignment behind the already async part | |
| 10:13:20 | sean-k-mooney | one thing i have been debating in my mind is shoudl we be removing all direct calls to the compute from the api | |
| 10:13:56 | lyarwood | casts are still fine right? | |
| 10:14:06 | lyarwood | or move everything behind conductor tasks | |
| 10:14:18 | lyarwood | the latter is lots of work | |
| 10:14:34 | sean-k-mooney | i was thinking evenutally everything goes though conductor but cast might be ok | |
| 10:15:04 | sean-k-mooney | the reasonign was so that the api does not need to speak the compute node rpc version | |
| 10:15:32 | sean-k-mooney | but ya thats a larger change then you need | |
| 10:16:12 | sean-k-mooney | having the api create the bdms and then cast to do the rest async shoudl be fine in this case | |
| 10:17:32 | gibi | melwitt: when you are up, I managed to make CI green on https://review.opendev.org/c/openstack/nova-specs/+/788243 (and get bauzas +2) | |
| 10:25:14 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: [stable-only] Fix ceph install in live migration hook https://review.opendev.org/c/openstack/nova/+/794000 | |
| 10:26:24 | sean-k-mooney | melwitt: gibi: quick question about consumer types. will there be an api that allows me to list all allocation by consumer types and optionall filter that | |
| 10:27:55 | sean-k-mooney | for filtering i was thinking by host, project and maybe consumer/vm id | |
| 10:28:52 | sean-k-mooney | the reason im asking is the mailing list thread related to premptible instnaces. | |
| 10:29:35 | sean-k-mooney | one of the suggtions i had was to explore using a "premptible" consumer type for premtiable instances | |
| 10:30:29 | sean-k-mooney | initally if we had a query api it would allow and external service to esaisly determing what instance could be killed via a placment query | |
| 10:31:42 | sean-k-mooney | but i coudl imaging later extending plamcnet so that it coudl optionalling include the capsity used by preamtiable instance in its allocation candates set with a new micorverion and query arg | |
| 10:33:15 | sean-k-mooney | not that im going to work on that any time soon but i just taught that would enable and intersting desgin direction to explore | |
| 10:37:14 | jkulik | lyarwood, sean-k-mooney: wouldnt "having the api create the bdms" mean, that we need distributed locking between the apis, so we don't run into races while creating those? | |
| 10:38:07 | sean-k-mooney | jkulik: we are creating them in the db so we can do a db level lock if we needed too | |
| 10:38:07 | lyarwood | We'd need rely on the DB ensuring there's only ever one active BDM for a given instance UUID and volume ID combo | |
| 10:38:42 | jkulik | is multi-attach possible for the same VM? | |
| 10:38:50 | lyarwood | not for the same instance no | |
| 10:38:57 | sean-k-mooney | jkulik: within the same vm no | |
| 10:39:54 | sean-k-mooney | if we need a lock we can take a row level lock on the instance uuid or instance and volumne uuid pair | |
| 10:40:20 | lyarwood | tbh a constraint should be enough | |
| 10:40:44 | lyarwood | at least that's what was agreed on in the past when this came up | |
| 10:41:05 | lyarwood | but that was in the context of us creating duplicate BDMs on failure | |
| 10:41:15 | lyarwood | *failure to attach | |
| 10:41:20 | lyarwood | that hasn't been an issue for a while now | |
| 10:41:36 | sean-k-mooney | ya there is no index in the bdm that we need to worry about is there | |
| 10:41:44 | jkulik | lyarwood: "for a while" - which version do I need to upgrade to? :D | |
| 10:42:06 | lyarwood | jkulik: this was way back in Kilo I think and only reported by one customer downstream | |
| 10:42:12 | lyarwood | I've never seen it since | |
| 10:42:37 | lyarwood | -> lunch brb | |
| 10:42:43 | jkulik | hm ... I'm on rocky and need to investigate why there are like 10 bdm entries (pretty much empty) and one filled with connection_info in my DB for multiple VMs | |
| 10:43:02 | jkulik | same (volume_id, instance_uuid) | |
| 10:44:24 | sean-k-mooney | the uniqe constrati on rocky is just the bdm uuid | |
| 10:44:26 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/db/sqlalchemy/models.py#L587 | |
| 10:45:01 | sean-k-mooney | thats also the case for master | |
| 10:45:04 | sean-k-mooney | lyarwood: ^ | |
| 10:45:26 | stephenfin | bauzas: Friendly reminder that the 'instance_type' cleanup series is still waiting your attention, if you have some time https://review.opendev.org/q/topic:%2522compute_rpc_6.0%2522+status:open | |
| 10:45:35 | sean-k-mooney | so the uniqe constraint in the db i currently not enough to prevent duplicats | |
| 10:45:37 | lyarwood | jkulik: yeah the others are marked as soft deleted (deleted=id etc) | |
| 10:45:52 | lyarwood | sean-k-mooney: yup this is a new constraint I'm talking about | |
| 10:46:01 | jkulik | no. they're not soft-deleted | |
| 10:46:01 | sean-k-mooney | oh ok | |
| 10:46:14 | sean-k-mooney | well currently the could be not marked as soft deleted | |
| 10:46:35 | sean-k-mooney | you can have duplicate entries with the current schema | |
| 10:46:54 | lyarwood | right but that's the race I was saying we hadn't seen reported for a while | |
| 10:47:09 | sean-k-mooney | ah ok | |
| 10:47:31 | lyarwood | jkulik: what does `openstack server volume list $instance` show? | |
| 10:47:53 | lyarwood | jkulik: the same duplicates? | |
| 10:48:06 | sean-k-mooney | well backportbale way to fix this is to add a db trigger on insert to enforce the constraint then on master update the uniqe constratint | |
| 10:48:08 | jkulik | lyarwood: let me find such an instance again. I cleaned them up already for the customer to continue | |
| 10:49:30 | lyarwood | jkulik: ah if this is outside of the instance creation flow there have been known and now resolved issues with things like live migration rollbacks etc that could cause this | |
| 10:49:40 | lyarwood | and/or volume attach flows I should say | |
| 10:50:44 | jkulik | we're on VMware, so no live-migration via Nova for us in rocky afaik. might be volume-attach flows then | |
| 10:51:57 | jkulik | if they're fixed, I'll have a look through the commits. let's see what I can dig up. thank you | |
| 10:52:46 | stephenfin | lyarwood: Replied on https://review.opendev.org/c/openstack/nova/+/794006 I wasn't hitting the code path you're expecting because I wasn't booting from volume | |
| 10:53:38 | opendevreview | sean mooney proposed openstack/nova-specs master: Add no user token when get Cyborg client https://review.opendev.org/c/openstack/nova-specs/+/787178 | |
| 10:53:39 | stephenfin | that's evidently a far less common request than boot from volume, which explains why nobody has (based on reports, anyway) hit this before | |
| 10:54:51 | jkulik | lyarwood: I didn't even know "server volume list" was a thing. cool. it shows both bdm entries for the instance I found | |
| 10:56:03 | sean-k-mooney | stephenfin: boot from volumen is often an optimasation fo boot form exsiting volume | |
| 10:56:21 | lyarwood | jkulik: cool, and `openstack server event list $instance` shows just attaching the volume etc? | |
| 10:56:56 | sean-k-mooney | i would expect the latter to maybe be more common for heat or horizon users where you are less likely to typo the uuid due to the indirection | |
| 10:57:54 | sean-k-mooney | stephenfin: booting form an existing volume used to be common if you were creating the volum using an iso | |
| 10:57:56 | jkulik | lyarwood: hm ... hard to say. k8s instance with lot's of attachments going on. but it looks like in this case the volume is in state error | |
| 10:58:13 | sean-k-mooney | stephenfin: where boot form volume the normal way will not do what you want it too | |
| 10:58:20 | stephenfin | sean-k-mooney: tbc, I'm not doing boot from volume here. I'm booting from an image and attaching an additional volume | |