| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-19 | |||
| 09:53:20 | bauzas | so the inventory is getting less | |
| 09:53:44 | sean-k-mooney | well thats good that we have a repoducer now | |
| 09:53:52 | sean-k-mooney | it should aid in fixing and backporting | |
| 09:56:57 | gibi | sean-k-mooney, bauzas: can we quickly discuss the force down requirement in https://review.opendev.org/c/openstack/nova/+/848886 ? | |
| 09:57:22 | gibi | sean-k-mooney: could you elaborate on the data corruption risk. is it depends on the task state? | |
| 09:59:04 | sean-k-mooney | well my ortinal concern was we were in the midel of doing an operation | |
| 09:59:12 | sean-k-mooney | it could be a snapshot for example | |
| 09:59:25 | sean-k-mooney | and depending on the backend we do strange thigns with say nfs | |
| 09:59:44 | sean-k-mooney | so im not sure how safe it is to always ignore it | |
| 10:00:34 | sean-k-mooney | like for nfs cinder volume snapshots we create a delta disks and update some paths in the xml and on the cidner side | |
| 10:00:49 | sean-k-mooney | if we evaucate in the midel of that i dont knwo what the sate will be | |
| 10:01:00 | sean-k-mooney | same for ceph i guess | |
| 10:01:06 | sean-k-mooney | in this case its just powering off | |
| 10:01:22 | sean-k-mooney | which shoudl be fine because we allow eveac with active vms | |
| 10:02:10 | gibi | ohh so the case is when the task was actually started by the compute service, then the compute service died. leaving a half uploaded snapshot or leaving a not fully updated geust xml behind | |
| 10:02:32 | sean-k-mooney | ya or if we were shelving ectra | |
| 10:02:49 | sean-k-mooney | basically if we are in the midel fo an operation im not sure what will happen | |
| 10:02:55 | sean-k-mooney | maybe its fine to evacuate | |
| 10:03:07 | sean-k-mooney | but we dont allwo that today | |
| 10:03:11 | gibi | so if that is the case then simply asking the admin to fence + force down before evac is not enough, the admin manually needs to clean up / repair things | |
| 10:03:29 | sean-k-mooney | ya i guess that is true | |
| 10:03:35 | gibi | right now force down only requires fencing | |
| 10:03:41 | sean-k-mooney | yes | |
| 10:03:46 | gibi | but going forward it will required a manual check by the admin | |
| 10:03:52 | gibi | and we need to be able to describe what to check | |
| 10:04:00 | sean-k-mooney | so i guess it really does not help in that respect | |
| 10:04:16 | sean-k-mooney | (force down) | |
| 10:04:40 | sean-k-mooney | i guess we are starting form this is blocked and you have to do reset-state | |
| 10:04:46 | sean-k-mooney | to evacuate | |
| 10:04:50 | gibi | yeah, it is convinient to push the responsibility to the admin by asking to force down, but we have to be able to tell the admin what to do before force down | |
| 10:05:47 | sean-k-mooney | if the node is down say psu exploded | |
| 10:05:59 | gibi | reset-state and force down is pretty similar in this regard, we ask the admin to do something and take over the burden of keeping the system consistent | |
| 10:06:00 | sean-k-mooney | i dont think there is much if anything the operator can do to clean up the state | |
| 10:06:31 | gibi | yeah | |
| 10:06:33 | sean-k-mooney | at least not without out lookign behind cinders back at the stroage | |
| 10:06:44 | sean-k-mooney | so maybe im overthinking it | |
| 10:06:55 | sean-k-mooney | but that is why i was suggesting force down | |
| 10:08:27 | gibi | we can suggest force down but then we have to change the definition of force down, as simple fencing the host is not enough any more | |
| 10:09:15 | sean-k-mooney | ya i think im coming around to your way of thinking and not overloading force-down | |
| 10:09:42 | sean-k-mooney | we perhaps should isntead just document that if the instance task state is not None | |
| 10:09:52 | sean-k-mooney | the operator may need to do addtional cleanup of the instance | |
| 10:10:15 | sean-k-mooney | e.g. remove a partial snapshot | |
| 10:10:17 | gibi | yepp | |
| 10:10:43 | gibi | we can keep the reset-state requirement as today if that helps against accidental evac | |
| 10:11:34 | sean-k-mooney | well i guess that is the choice we have to make. if we think the timeout is suffeicnet as it has been in the past | |
| 10:11:49 | sean-k-mooney | then we can proceed with the change and just drop the force down requirement and add some extra docs | |
| 10:12:06 | sean-k-mooney | otherwise yes we can leave it as it is today with reset state | |
| 11:13:18 | bauzas | OK, I need to go lunching, but I think I see the problem with the mdev names | |
| 11:13:49 | bauzas | it creates an exception when we run the periodic RT method for updating | |
| 11:14:13 | sean-k-mooney | ack that is what i was expecting woudl happen | |
| 11:14:34 | sean-k-mooney | either an excption or the list would be empty | |
| 11:14:45 | sean-k-mooney | in either case resulitng int placment getting out of sync | |
| 11:14:49 | sean-k-mooney | and preventing reuse | |
| 11:27:28 | opendevreview | Merged openstack/nova master: libvirt: Ignore LibvirtConfigObject kwargs https://review.opendev.org/c/openstack/nova/+/830644 | |
| 11:47:40 | opendevreview | Merged openstack/nova master: libvirt: Remove unnecessary TODO https://review.opendev.org/c/openstack/nova/+/830645 | |
| 11:54:24 | sean-k-mooney | stephenfin: i added https://review.opendev.org/c/openstack/nova-specs/+/849488 to open discussion to ask for the spec freeze exception | |
| 11:55:18 | sean-k-mooney | stephenfin: so lets defer the +w to sylvain so they can either -2 it if we reject the exctpion or +w if we accept assuming they agree with the spec content | |
| 12:06:07 | stephenfin | sounds good | |
| 12:08:27 | sean-k-mooney | stephenfin: remind me you orgianlly wanted to truncate the displayname to set the hostname right | |
| 12:08:44 | sean-k-mooney | rather then normalise | |
| 12:10:17 | sean-k-mooney | im stongly considering if we shoudl have done neither and added support for fqdns when talking to neutron by doing the truncation there | |
| 12:10:53 | stephenfin | oh, I've no idea /o\ I'd have to go check the spec/patches | |
| 12:11:08 | sean-k-mooney | ya not really imporant now i guess | |
| 12:11:30 | sean-k-mooney | context is https://review.opendev.org/c/openstack/nova-specs/+/849765 and whetere or not we should do this in nova | |
| 12:14:44 | opendevreview | sean mooney proposed openstack/nova-specs master: Revert "Configurable instance domains" https://review.opendev.org/c/openstack/nova-specs/+/850048 | |
| 12:17:06 | sean-k-mooney | dansmith: ^ ill wait for the discusion in the nova team meeting to determin if i shoudl repopose the spec un a spec freeze exception or if we defer to AA | |
| 12:17:30 | sean-k-mooney | in which case we dont need to rush to reopen the review and we can wait for artom to return form pto | |
| 12:32:10 | opendevreview | sean mooney proposed openstack/nova-specs master: Revert "Configurable instance domains" https://review.opendev.org/c/openstack/nova-specs/+/850048 | |
| 12:32:25 | sean-k-mooney | ^ less typos and better commit | |
| 12:46:27 | opendevreview | Stephen Finucane proposed openstack/nova master: Use unittest.mock instead of third party mock https://review.opendev.org/c/openstack/nova/+/714676 | |
| 12:46:28 | opendevreview | Stephen Finucane proposed openstack/nova master: Remove the PowerVM driver https://review.opendev.org/c/openstack/nova/+/850346 | |
| 13:28:15 | opendevreview | ribaudr proposed openstack/nova master: Allow unshelve to a specific host (REST API part) https://review.opendev.org/c/openstack/nova/+/845897 | |
| 13:46:01 | opendevreview | Merged openstack/nova-specs master: Revert "Configurable instance domains" https://review.opendev.org/c/openstack/nova-specs/+/850048 | |
| 14:11:44 | opendevreview | sean mooney proposed openstack/nova-specs master: Revert "Revert "Configurable instance domains"" https://review.opendev.org/c/openstack/nova-specs/+/850352 | |
| 14:12:38 | sean-k-mooney | ok i have mad the instnace.domain -> instance.dns_domain change and tried to call out the open issues in ^ | |
| 15:23:16 | bauzas | I'm under deep water but we'll have our weekly meeting | |
| 15:23:24 | bauzas | 37 mins from now here | |
| 15:23:29 | bauzas | I'll prepare the agenda | |
| 15:27:08 | Uggla | Went out for a bike ride to get my car from the garage. It is really hot today. | |
| 15:33:25 | admin1 | hi .. i have cpu_allocation_ratio is at 4.0, but it refuses to go above the physical threads .. nova scheduler reporting: There was a conflict when trying to complete your request. Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider 'UUID '. The requested amount would exceed the capacity. | |
| 15:33:54 | admin1 | if i want to deploy something below the actual vcpus, it works .. but it does not go above the physical threads | |
| 15:34:00 | sean-k-mooney | admin1: you canno thave a singel allcotion that excced the numa of actual cpus | |
| 15:34:18 | admin1 | sorry .. what does that mean :) | |
| 15:34:33 | sean-k-mooney | we do not allow vms to over subscibel against themselves | |
| 15:34:51 | sean-k-mooney | so 1 vm can never have more vcpu then the host has | |
| 15:35:00 | sean-k-mooney | the allocation ratio is not related too that | |
| 15:35:30 | admin1 | i have 40 cpus ( physical threads) .. and ratio is 4.0 .. but i can only have 2 instance of 16 vcpu there | |
| 15:35:53 | sean-k-mooney | hum you shoudl be able to boot more | |
| 15:36:05 | sean-k-mooney | so the total on the inventory is 40 | |
| 15:36:09 | admin1 | yes | |
| 15:36:12 | sean-k-mooney | and used is 32 | |
| 15:36:15 | admin1 | right | |
| 15:36:21 | sean-k-mooney | what version of mariadb are you useing | |
| 15:36:45 | sean-k-mooney | you might be hitting a mariadb bug that was reported on the mailing list last week | |
| 15:36:53 | admin1 | Server version: 10.6.5-MariaDB-1:10.6.5+maria~focal-log mariadb.org binary distribution | |
| 15:36:54 | sean-k-mooney | we are posibly seeing the same bug downstream | |
| 15:37:17 | sean-k-mooney | admin1: yep that version is apprently broken and its fixed in 10.6.8 i belive | |
| 15:37:58 | sean-k-mooney | admin1: see this thread https://lists.openstack.org/pipermail/openstack-discuss/2022-July/029536.html | |
| 15:38:23 | admin1 | sean-k-mooney thanks for the direction | |
| 16:00:21 | bauzas | #startmeeting nova | |