| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-15 | |||
| 12:31:48 | sean-k-mooney | brinzhang: rpc backports cant realy be done | |
| 12:33:05 | brinzhang | sean-k-mooney: I know, but if I upgrade the rocky release to ussuri, that will be some impact than backport, so I just want to backport it | |
| 12:33:27 | brinzhang | but encountered the rpcverion invalid issue | |
| 12:33:34 | sean-k-mooney | brinzhang yes | |
| 12:33:57 | sean-k-mooney | you cant backport rpc change because it will break upgrades and rolling deployments | |
| 12:34:14 | sean-k-mooney | we have tried downstream and it really did not go well | |
| 12:34:20 | brinzhang | does it need to change the oslo.messaging to adopt 5.1? | |
| 12:35:17 | brinzhang | I got +1 on the basis of the rpc version of the Rocky version, which is not a complete backport | |
| 12:35:38 | sean-k-mooney | +1 from who? | |
| 12:36:10 | sean-k-mooney | we should not attempt to backport this upstream | |
| 12:36:35 | brinzhang | +1 from rocky rpc version 5.0-->5.1, and change the target_compute version from 35 to 36 | |
| 12:37:32 | brinzhang | I looked into the service db table, the node compute can be 36, but when we restart the compute service,the rpc version also in 5.0 | |
| 12:38:33 | brinzhang | sean-k-mooney: Backport is a last resort | |
| 12:39:25 | brinzhang | base on https://github.com/openstack/nova/blob/stable/rocky/nova/compute/rpcapi.py#L351, +1 for the rpcversion | |
| 12:40:01 | brinzhang | refer this patch: https://review.opendev.org/#/c/634606/86 | |
| 12:44:53 | gibi | brinzhang: I've never tried backporting an RPC change. | |
| 12:45:32 | sean-k-mooney | gibi: stephenfin has | |
| 12:45:37 | gibi | I did assisted backporting API changes downstream, there we simply put the new API into the latest microversion of that branch. It is ugly but it works | |
| 12:46:16 | sean-k-mooney | that is ya less then good if you have to ship it to others | |
| 12:46:17 | brinzhang | gibi: I want to know what key points I have overlooked. This method of incrementing the version number based on the rocky version is the same as adding a feature to the master, but why doesn't it take effect? | |
| 12:46:41 | sean-k-mooney | our rule is if it has object,db,api or rpc change its not backported | |
| 12:47:57 | brinzhang | the rules are clear :D | |
| 12:48:20 | sean-k-mooney | am have you pinned your rpc version in the config | |
| 12:48:43 | gibi | sean-k-mooney: yeah, it was a compromise where the necessary risk and future cost was accepted (and I don't have to touch it so I'm fine with it) | |
| 12:48:53 | brinzhang | but in theoretically, my thought can pass, but it is not | |
| 12:49:17 | sean-k-mooney | gibi: we have considered doing the same for api changes but decied not to for feature discovery | |
| 12:49:55 | gibi | brinzhang: might be your problem related to the RPC version alias on the given branch (simply guessing here) | |
| 12:50:02 | sean-k-mooney | e.g. we did not want ot create interoperabity issue between clouds | |
| 12:50:25 | sean-k-mooney | gibi: yep that is want i was wondering for the config | |
| 12:50:28 | sean-k-mooney | more or less | |
| 12:50:45 | sean-k-mooney | e.g. is it pinned to a version and or an alias | |
| 12:51:56 | brinzhang | gibi: alias? what does this mean? | |
| 12:52:14 | brinzhang | RPC version alias? | |
| 12:52:43 | gibi | brinzhang: https://docs.openstack.org/nova/latest/user/upgrade.html#concepts | |
| 12:53:11 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/rpcapi.py#L354-L364 | |
| 12:54:20 | brinzhang | sean-k-mooney: we dont change the VERSION_ALIASES | |
| 12:54:43 | brinzhang | it is as the same as your paste | |
| 12:54:57 | sean-k-mooney | yes but what do you have your rpc version set to in the config | |
| 12:55:22 | sean-k-mooney | blank, auto, an alias or a value | |
| 12:55:57 | brinzhang | the default, because we dont config the rpcversion in conf file | |
| 12:56:27 | sean-k-mooney | so i think that will default to the newest version supported by the current node | |
| 12:56:48 | sean-k-mooney | meaing if you have not applie the change to all serivce some will be 5.0 and other will be 5.1 | |
| 12:57:06 | sean-k-mooney | auto i think figures out thet max supported version for all nodes | |
| 12:57:34 | sean-k-mooney | for the numa migration at least both comptue nodes and the conductor would need 5.1 | |
| 12:58:00 | brinzhang | sean-k-mooney: yeah, I can try your suggestion, I changed two nodes's code in my 3 nodes cluster | |
| 12:58:30 | brinzhang | but stoped the nova-* services in another node | |
| 12:59:12 | sean-k-mooney | i think we store the version in the db somewhere | |
| 12:59:46 | sean-k-mooney | so stopping it might not be enough dansmith is who i normally ask about this becasue i generally try to avoid it | |
| 12:59:59 | brinzhang | I just find the compute version in service db table | |
| 13:00:53 | sean-k-mooney | this is still the video i go to to understand this stuff https://www.youtube.com/watch?v=wVXiw09XP04 i wonder how much of that still applies | |
| 13:01:47 | brinzhang | https://github.com/openstack/nova/blob/stable/rocky/nova/conf/rpc.py this conf file seems not contain your said auto, default value | |
| 13:02:36 | brinzhang | yeah, I will see the vedio tomorrow, need to jump the firewall | |
| 13:02:38 | sean-k-mooney | auto is not the default | |
| 13:02:58 | brinzhang | if I want to use auto, how to config? | |
| 13:03:04 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/conf/upgrade_levels.py | |
| 13:03:10 | sean-k-mooney | but also that is the file you want | |
| 13:03:37 | sean-k-mooney | the default is unest which means use the latest vers this code knows | |
| 13:03:46 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/conf/upgrade_levels.py#L55-L61 | |
| 13:04:35 | sean-k-mooney | [upgrade_level]/compute=auto | |
| 13:04:51 | brinzhang | got it, thanks | |
| 13:04:56 | sean-k-mooney | in this case you want 5.1 | |
| 13:05:04 | sean-k-mooney | so you can just set that | |
| 13:05:21 | sean-k-mooney | you also need to do it for the conductor | |
| 13:05:30 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/conf/upgrade_levels.py#L117 | |
| 13:05:51 | brinzhang | yeah, I will try, two ways: 1. set [upgrade_level]/compute=auto in nova compute node, and then to do live migration, | |
| 13:06:07 | sean-k-mooney | you need to set it for both compute and conductor | |
| 13:06:13 | brinzhang | 2. replaced all nodes's code of nova, and then try again | |
| 13:06:25 | brinzhang | ack | |
| 13:06:43 | sean-k-mooney | i assume you have also backported all the ovo changes | |
| 13:07:10 | sean-k-mooney | e.g. the change to the migrate_data object to store the numa toplogy | |
| 13:08:15 | brinzhang | yes, I was backported all the bp's patches | |
| 13:08:48 | sean-k-mooney | there have been some bugfixes too but it more or less shoudl be fine | |
| 13:09:04 | sean-k-mooney | brinzhang: you are aware you can manually live migrate numa instance before too right | |
| 13:09:12 | sean-k-mooney | we just dont recaluate the xml | |
| 13:09:31 | sean-k-mooney | so if you have 1 free compute node you can live mgirate without thse backports to do a rolling upgrade | |
| 13:10:10 | sean-k-mooney | you just need to ensure the cpus and hugepages exctra are free on the dest | |
| 13:10:27 | brinzhang | yes | |
| 13:11:40 | brinzhang | I will use two node to have a test, at leat two node, because I should ensure this feature is fine, otherwise, it will have some risk, it's not allow | |
| 13:12:33 | brinzhang | Later I will search for related bug fixes and backport | |
| 13:13:00 | sean-k-mooney | have you also backported sriov live migration i assume not | |
| 13:13:55 | brinzhang | sriov live migration we completed it by ourself, dont need to backport ^^ | |
| 13:14:34 | brinzhang | If we upgrade in future, and we will use the merged codes too | |
| 13:14:45 | sean-k-mooney | the reason i bring that up is numa migratio nmerged after sriov and at least 1 of the patches in the numa serises is adding compatiablty with sriov live migrfation | |
| 13:15:07 | sean-k-mooney | your going from rocky to train right | |
| 13:15:19 | brinzhang | yes, it contained, I was found your point | |
| 13:15:38 | sean-k-mooney | train is the verssion that had both so once your on train you shoudl be fine | |
| 13:15:41 | brinzhang | maybe from rocky to ussuri | |
| 13:16:27 | brinzhang | because we also want to use the pmem feature too ^ | |
| 13:17:16 | brinzhang | sean-k-mooney: thanks for your help | |
| 13:17:43 | brinzhang | I will go home now, have a holiday sean-k-mooney, gibi :D | |
| 13:17:56 | gibi | brinzhang: o/ | |
| 13:41:03 | dansmith | sean-k-mooney: compute=auto will try to select the version supported by the oldest compute service in the system, based on service_version | |
| 13:41:43 | sean-k-mooney | dansmith:yes | |
| 13:41:53 | sean-k-mooney | the max version suspport by the oldest | |
| 13:42:03 | sean-k-mooney | which is why i was saying to use 5.1 | |
| 13:42:07 | sean-k-mooney | not auto | |
| 13:42:26 | sean-k-mooney | since brinzhang is not update all the nodes | |
| 13:42:29 | dansmith | ah, you said [05:56:27] <sean-k-mooney>so i think that will default to the newest version supported by the current node | |
| 13:42:38 | dansmith | but you meant that for if it's unset or something? | |
| 13:42:48 | sean-k-mooney | yes unset | |