| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-13 | |||
| 15:32:50 | TheJulia | unless the conductor is watching like a hawk and overriding the table contents | |
| 15:33:04 | sean-k-mooney | in the gneral case yes. if its a littel more protacted then it would | |
| 15:33:24 | TheJulia | yes, but at which point that compute is out of service or unable to be used | |
| 15:34:10 | sean-k-mooney | yep | |
| 15:34:11 | TheJulia | I know in ironic that we don't drop it from the working list until after 3 failures have occured, but I'd need to check the code in nova | |
| 15:34:24 | sean-k-mooney | i think its similar | |
| 15:34:39 | TheJulia | which means it wouldn't be disqualified I think until 90 seconds have actually passed | |
| 15:34:39 | sean-k-mooney | we dont mark it ad down until multiple heart beats are missed | |
| 15:34:42 | sean-k-mooney | i think its also 3 | |
| 15:34:52 | sean-k-mooney | ya | |
| 15:35:00 | sean-k-mooney | how long does rebalancing typically take | |
| 15:35:17 | sean-k-mooney | is it second or minutes? i assume the former | |
| 15:35:39 | TheJulia | seconds for the actual mapping at worst, the actual updates and iteration through is the painful part | |
| 15:35:58 | sean-k-mooney | without using disables as a distiubed lock | |
| 15:36:11 | sean-k-mooney | how do you prevent the compute when it compes back form racing iwth the reblance | |
| 15:36:19 | sean-k-mooney | that was the other usecase for it | |
| 15:36:25 | TheJulia | I don't think we've ever measured a startup rebalance on a large site in ironic but I've generally heard of a couple minutes | |
| 15:37:15 | TheJulia | sean-k-mooney: the code, as I remember it would check to see if the prior compute is back and then breakout of the rebalanance apply loop if so | |
| 15:37:50 | TheJulia | since surely in a little bit, it was going to undo some of what it just did since the hash ring is back to what it was prior to the failure | |
| 15:38:08 | sean-k-mooney | that makes sense | |
| 15:38:26 | TheJulia | a "oh, the universe just changed on us. stop what we're doing!" | |
| 15:38:31 | TheJulia | check | |
| 15:38:54 | sean-k-mooney | yep but it need to also roleback so liekly everthing need to be in one big transaction | |
| 15:39:07 | sean-k-mooney | so we either get the rebalanced sate or the old state | |
| 15:43:48 | TheJulia | a giant transaction would acutally harm the ability to understand what is going on because it also checks current record state so if that is effectively hiding in a giant transaction which will take a minute or two to commit, then we're introducing ourselves to more issues | |
| 15:44:15 | sean-k-mooney | it should not take minutes to commit | |
| 15:44:32 | sean-k-mooney | we shoudl just compute the desired endstate then comiit the change in one trasaction | |
| 15:44:47 | TheJulia | a couple thousand specific column field updates in a heavily used table? | |
| 15:44:48 | sean-k-mooney | you just need to use a case statement to update the instance.host on all instance in one go | |
| 15:45:02 | TheJulia | eh... then again performance has changed drastically since the days I did that often manually | |
| 15:46:03 | sean-k-mooney | so i think its just one colume on one db table with many rows | |
| 15:46:11 | sean-k-mooney | that shoudl be quick | |
| 15:46:40 | TheJulia | eh, last time I did something like that manually like 4 thousand rows in a 100k row table it was a couple minutes, but again, that was like a decade ago | |
| 15:47:48 | TheJulia | Anyway, I'd need to double check a few things but a transaction, at least to me seems more harmful in that the compute node's rebalance is hidden state so we could get a bunch of conflicting transactions pile up, all fail past the first one, and trash to try and correct some of it as time goes on consistency wise | |
| 15:49:41 | sean-k-mooney | well if we dont have a transaction you will need to role back the partially reblanced state | |
| 15:49:52 | sean-k-mooney | manually right | |
| 15:50:07 | sean-k-mooney | or trigger a full rebalnace again | |
| 15:50:18 | sean-k-mooney | to even out the load on the compute services | |
| 15:50:45 | opendevreview | Merged openstack/nova-specs master: add per process healthcheck spec https://review.opendev.org/c/openstack/nova-specs/+/821279 | |
| 15:52:32 | TheJulia | well... a transaction could work in try to generate the updates and then only check if things have changed before committing it if things have not changed, but I'd need to dig into the dbapi internals since to do so likely sounds like rpc functionality would be required to batch it all up properly outside of the normal transactions oslo_db drives | |
| 15:52:40 | TheJulia | since we're doing it one at a time instead of a bulk change | |
| 15:54:03 | sean-k-mooney | we would need to have a db method for this yes not just loop over the isntance set the filed and call save | |
| 15:54:09 | TheJulia | I guess I'm worried about trying to overthink it and then over engineer it, or if a model of eventual consistency is a happier place to be. I'm very much on the eventual consistency mindset since for ironic, it *really* doesn't matter which node proxies the request as long as it is online | |
| 15:54:51 | TheJulia | it all matters for rpc request routing ultimately | |
| 15:54:52 | sean-k-mooney | i wonder if for ironic it would be better to have a single shared topic queue | |
| 15:55:16 | TheJulia | what would that change/gain us in this situation? | |
| 15:55:17 | sean-k-mooney | so that all the compute shared one queue and any of them could deque it | |
| 15:55:34 | sean-k-mooney | it would mena the isntance.host woudl not matter anymore | |
| 15:55:35 | TheJulia | hmm, that could entirely do away with the hash ring | |
| 15:55:48 | TheJulia | well, kind of | |
| 15:56:20 | TheJulia | hmmmm it would have to know it is ironic | |
| 15:56:29 | TheJulia | but that could actually be navigated upgrade wise too | |
| 15:56:48 | TheJulia | that doesn't fix the UX issues which are ultimately bugs in past releases though | |
| 15:56:55 | sean-k-mooney | yes but we do know the hypervior type at least on the ocmpute node side not sure about the instnace object | |
| 15:57:21 | TheJulia | I don't think it is on the instance object | |
| 15:57:28 | TheJulia | but again, I've not looked at its structure in a while | |
| 15:58:00 | sean-k-mooney | well the way to hack it is to decied instance.host would always be set to "ironic" or simiarl for ironci contoled instnaces | |
| 15:58:23 | sean-k-mooney | anyway the impartnat thing is you still care about fixing this issue | |
| 15:58:30 | sean-k-mooney | but dont curretnly have time to work on it | |
| 15:59:40 | sean-k-mooney | so we shoudl consider if we "redhat comptue team" have capsity to help this cycle or next | |
| 16:00:26 | sean-k-mooney | we coudl land your patch as is but im not sure long term its the best approch but it would stop the bleeding in the short term | |
| 16:00:54 | sean-k-mooney | which is the whole perferect is the enemy of good enough argument | |
| 16:40:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: trigger nova-next with new tempest test https://review.opendev.org/c/openstack/nova/+/824607 | |
| 16:43:39 | TheJulia | sean-k-mooney: yeah. :( | |
| 16:43:58 | TheJulia | sorry, been distracted looking at a blocker issue | |
| 16:53:55 | admin1 | hi all .. how to figure out what puts messages in versioned_notifications.info ..but there are no consumers | |
| 16:54:06 | admin1 | so the queue just grows and grows .. and then i have to manually delete it | |
| 16:54:55 | gibi | admin1: you can disable notifications | |
| 16:55:09 | admin1 | gibi, from where/how ? | |
| 16:55:15 | gibi | sec... | |
| 16:56:30 | gibi | admin1: https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html#oslo_messaging_notifications.driver | |
| 16:56:39 | gibi | admin1: so in the nova service configuration files | |
| 16:56:53 | gibi | admin1: set [oslo_messaging_notifications]driver=noop | |
| 16:59:07 | sean-k-mooney | admin1: its disabled by default | |
| 16:59:15 | gibi | sean-k-mooney: I don't think so | |
| 16:59:24 | sean-k-mooney | the noop driver is the defult | |
| 16:59:25 | gibi | sean-k-mooney: I think the default is messaging_v2 | |
| 16:59:31 | sean-k-mooney | i dont think so | |
| 16:59:37 | gibi | hm | |
| 16:59:40 | gibi | interesting | |
| 16:59:46 | gibi | according to the doc you are right | |
| 16:59:55 | gibi | I alwas remembered it is enable by default | |
| 17:00:01 | sean-k-mooney | devstack enables it by default | |
| 17:00:27 | sean-k-mooney | and it also defaluts to unversioned if i remember | |
| 17:00:37 | gibi | yes, the unversioned is the default | |
| 17:00:44 | gibi | we never switched it to versioned | |
| 17:00:52 | sean-k-mooney | we really should | |
| 17:01:05 | sean-k-mooney | and eventurlly remove the deprecated unversioned notifications | |
| 17:01:13 | sean-k-mooney | they have been deprecated since mitaka | |
| 17:01:26 | gibi | sean-k-mooney: the problem that there are openstack services using the unversioned | |
| 17:01:33 | gibi | so we would migrate them first | |
| 17:01:57 | sean-k-mooney | ya but honestly we should propose that a a cross projectr goal | |
| 17:02:28 | sean-k-mooney | the problem really haing people to do the work | |
| 17:02:50 | sean-k-mooney | maybe a topic of next ptg | |
| 17:02:54 | sean-k-mooney | do we know which ones rely on it | |
| 17:03:16 | gibi | I have to dig | |
| 17:03:28 | gibi | ceilometer is one I'm sure | |
| 17:03:31 | sean-k-mooney | i support cilometer, cloud kitty, heat or masikari are the only ones that might be impacted | |
| 17:03:39 | sean-k-mooney | maybe watcher | |
| 17:04:25 | sean-k-mooney | as long as we are not adding any new unversion notification i guess it does not hurt use too much | |