| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-05 | |||
| 15:20:24 | sean-k-mooney | i think i reverted that in my rebuild patch for a reason | |
| 15:21:31 | gibi | I think earlier _rebuild_cyborg_arq was extended to handle unshelve too but I asked not to do that | |
| 15:23:08 | sean-k-mooney | oh they are moveing wherre we get the nodename | |
| 15:23:51 | gibi | sean-k-mooney: in different action we have different source of the node | |
| 15:23:51 | sean-k-mooney | yah i suspec there is a second caller to _create_and_bind_arq_for_instance | |
| 15:24:04 | sean-k-mooney | that only has the hostname not the host | |
| 15:24:48 | sean-k-mooney | here https://review.opendev.org/c/openstack/nova/+/729563/26/nova/conductor/manager.py#1251 | |
| 15:31:47 | sean-k-mooney | gibi: ya im not sure im a fan of folding _rebuild_cyborg_arq into the calling function | |
| 15:32:00 | gibi | that could be a comment | |
| 15:32:26 | sean-k-mooney | the other changes all look correct to me in the conductor manager however | |
| 15:33:50 | sean-k-mooney | well it only had one caller | |
| 15:34:21 | sean-k-mooney | and i intoduced it so ti could be reused for the other opertion so if we are not doing that removing it makes sense | |
| 15:34:41 | sean-k-mooney | its just the calling function is already quite large so its was nice to have it broken out | |
| 15:36:32 | gibi | yeah | |
| 15:36:41 | gibi | they tried to reuse it but that would neede a new flag | |
| 15:36:46 | gibi | so I was against that | |
| 15:36:54 | gibi | but we can keep it the hlper | |
| 15:36:56 | gibi | helper | |
| 15:44:12 | sean-k-mooney | its a nice to have rather then anything functional so not enough to -1 over | |
| 17:12:42 | openstackgerrit | Dan Smith proposed openstack/nova stable/victoria: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/761923 | |
| 17:13:19 | dansmith | gibi: if you will double-check my edits here I can just +2+W this as I just fixed nits ^ | |
| 17:14:42 | gibi | dansmith: looking | |
| 17:16:24 | gibi | dansmith: thanks, looks good to me | |
| 17:20:38 | dansmith | ack | |
| 23:59:24 | openstackgerrit | Merged openstack/nova master: Improving the description for unshelve request body https://review.opendev.org/c/openstack/nova/+/767251 | |
| #openstack-nova - 2021-01-06 | |||
| 01:17:27 | openstackgerrit | melanie witt proposed openstack/nova master: Enable test_volume_backed_live_migration in tempest https://review.opendev.org/c/openstack/nova/+/528104 | |
| 01:30:11 | brinzhang0 | bauzas, sean-k-mooney: I have replied the cyborg shelve/unshleve patch, pls review again, and I think there is no need to update in, one comments need to move the comments, and I will addressed in the followed up patch | |
| 06:53:47 | openstackgerrit | Mamduh proposed openstack/os-vif stable/queens: Refactor code of linux_net to more cleaner and increase performace https://review.opendev.org/c/openstack/os-vif/+/765941 | |
| 06:53:48 | openstackgerrit | Mamduh proposed openstack/os-vif stable/queens: Fix - os-vif fails to get the correct UpLink Representor https://review.opendev.org/c/openstack/os-vif/+/765983 | |
| 08:21:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove dead code from SchedulerReportClient https://review.opendev.org/c/openstack/nova/+/769467 | |
| 09:42:14 | alexe9191 | Hello There, | |
| 09:42:50 | alexe9191 | I was wondering if I can run a newer version of nova-api/scheduler/conductor on an older version of nova-compute. Nova-compute would be kili for instance and the newer version of the API services would be on Pike or Rocky for instance? | |
| 09:44:47 | gibi | alexe9191: nova keeps compatibility between version N+1 controller and version N compute service | |
| 09:45:28 | gibi | so you can have Rocky controller services and Pike compute services | |
| 09:45:41 | gibi | sorry, so you can have Rocky controller services and Queens compute serivces | |
| 09:46:14 | gibi | as Q + 1 = R | |
| 09:46:18 | alexe9191 | interesting, what out of curiosity. Does this also work the other way around? | |
| 09:46:30 | alexe9191 | so nova-compute R and nova-api Q ? | |
| 09:49:19 | gibi | no, the controller services need to be upgraded first | |
| 09:50:17 | alexe9191 | I am also guessing that I can't immediately upgrade the nova-compute from kilo to rocky in one go? This also has to be done in a rolling fashion? So basically, K -> L -> etc > R on the nova-computes ? | |
| 09:51:38 | gibi | alexe9191: what we test is upstream is rolling upgrade as you guessed. There are some support for fast forward upgrade where you move more than one release forward in single step | |
| 09:51:52 | gibi | https://wiki.openstack.org/wiki/Fast_forward_upgrades | |
| 09:54:36 | alexe9191 | I honestly fail to understand that a little bit as the documentation is not really clear on fast forward upgrades | |
| 09:55:41 | sean-k-mooney | alexe9191: fast forward upgrades are not really a feature of the comonet project more an installer feature that the compoent project like nova try to not intentionally break | |
| 09:55:58 | sean-k-mooney | stictly speaking nova only support n to n+1 | |
| 09:56:19 | alexe9191 | So basically this fast forward *feature* is more of a wrapper around what nova support ? | |
| 09:57:05 | sean-k-mooney | not quite it a way of runnign all the db updates and migration for the intermidat release so that you can skip them | |
| 09:57:42 | sean-k-mooney | on the nova side we try to keep that logic around for multiple release so that if you skip 3 version of the compute agent it will still have the compatibley code | |
| 09:58:09 | sean-k-mooney | so nova "supports" fast forward upgrades | |
| 09:58:18 | alexe9191 | So for instance running the DB migration scripts from newton while one is running Kilo ? | |
| 09:58:33 | sean-k-mooney | by keeping the compat code for several release beyond when it was striclty reuired | |
| 09:59:13 | sean-k-mooney | when you do an FFU there is a period where the contolplane is in accessable. | |
| 09:59:41 | sean-k-mooney | basiclly you move the contoler services to newton while the computes are on kilo | |
| 09:59:48 | sean-k-mooney | then you move the compute to newton | |
| 10:00:11 | sean-k-mooney | when you do that in principal the newton contoler cannot manage the kilo compute but the vms keep running | |
| 10:00:35 | sean-k-mooney | when you bring the compute back up to newton manageablity is restored | |
| 10:01:22 | sean-k-mooney | in pratice if there has been no RPC major version bump between the two version you have deploy the contoler could tehoretically mange compute more then 1 releasee old | |
| 10:01:27 | sean-k-mooney | but we dont test that | |
| 10:01:40 | alexe9191 | ok, and rolling upgrade being moving everything at once from one release to the other, both the APi & the nova-compute agents, K, L, N, ... | |
| 10:01:46 | sean-k-mooney | there is typically 5-6 release betwwen major rpc version bumps | |
| 10:02:32 | alexe9191 | That's very helpful to know! | |
| 10:02:50 | sean-k-mooney | alexe9191: rolling upgrades are where you move the contolplane in one go to n+1, then move the computes in a "rolling" or incremental fashion 1 by 1 | |
| 10:03:31 | sean-k-mooney | rolling upgrades allow you to maintain managmeablity of the computes with mixed verions | |
| 10:03:41 | sean-k-mooney | but we only test a delta of 1 verions | |
| 10:04:11 | sean-k-mooney | we write the code such that it could be a larger delta but the test matix is too large for use to offialy support that | |
| 10:04:12 | alexe9191 | Right, so if upgrading K to L. I could basically roll the upgrade to the compute nodes one rack at a time from K to L | |
| 10:04:28 | sean-k-mooney | yes | |
| 10:04:47 | alexe9191 | So in theory one could do N+2 but that's not tested. | |
| 10:05:28 | sean-k-mooney | yes in princiapl n->n+2 should work provided there is not an rpc major version bump in between | |
| 10:05:47 | sean-k-mooney | but really you would be the first to test that | |
| 10:06:10 | alexe9191 | I'd rather stick to what you guys are doing upstream in that case. | |
| 10:06:13 | sean-k-mooney | we have been on RPC version 5.X for about 5 releases now | |
| 10:06:44 | sean-k-mooney | alexe9191: ya if this is a prod deployment that is likely your safest option | |
| 10:06:49 | alexe9191 | I am guessing there are major changes between K and R in RPC versions, is that documented in the release note? | |
| 10:07:05 | alexe9191 | Yes, which sounds that doing the rolling-upgrades is the safest supported solution. | |
| 10:08:08 | sean-k-mooney | i think it would be in the release notes but i can quickly check the code | |
| 10:08:38 | alexe9191 | Fantastic, if I could also know which file you are checking in the source tree that would be great as well. | |
| 10:09:07 | sean-k-mooney | we have each verion bump documented with a comment https://github.com/openstack/nova/blame/master/nova/compute/rpcapi.py#L76 | |
| 10:09:37 | sean-k-mooney | https://github.com/openstack/nova/blame/master/nova/compute/rpcapi.py#L385 | |
| 10:09:42 | alexe9191 | Awesome! | |
| 10:09:45 | sean-k-mooney | the aliase are the quickest way to check | |
| 10:10:23 | alexe9191 | Interesting, so K to P is on the same Major RPC release. | |
| 10:10:32 | sean-k-mooney | so kilo was 4.0 | |
| 10:10:39 | sean-k-mooney | rocky was 5.0 | |
| 10:10:57 | sean-k-mooney | technically 5.0 is the ame as 4.22 | |
| 10:11:46 | sean-k-mooney | so if rocky still has the 4.22 compat code then you could pin the rpc version to 4.0 in principal | |
| 10:11:56 | sean-k-mooney | but it proably will have bugs | |
| 10:12:36 | alexe9191 | This is really helpful! | |
| 10:13:35 | sean-k-mooney | honestly i doubt we actully manged to keep perfect rpc compate and i suspect rocky has had the 4.x compat code removed | |
| 10:14:31 | sean-k-mooney | but looking quickly it looks like the compat code was maybe removed in stine | |
| 10:14:58 | sean-k-mooney | without testign it first however i would not bet on it it fully working | |
| 10:15:07 | alexe9191 | If I am not mistaken, this could mean that in theory I could run Newton API on kilo nova compute. Since the RPC API is on the same major version. | |
| 10:15:39 | sean-k-mooney | in theory yes | |
| 10:16:02 | sean-k-mooney | in practice you would likely be the first to try that and you would have to let us know if it works :) | |
| 10:16:27 | alexe9191 | I will most definitely do that :) | |
| 10:16:33 | sean-k-mooney | you would pin the RPC version of the newton contolplane to 4.0 | |
| 10:16:49 | alexe9191 | This should be done in nova.conf I could imagine ? | |
| 10:16:50 | sean-k-mooney | you can do that via the upgrade levels in the nova.conf | |
| 10:19:02 | alexe9191 | I am gonna let you guys know once I test that! Thanks a lot for all of the helpful information. | |
| 10:19:44 | bauzas | sean-k-mooney: alexe9191: well, Kilo is very prehistoric | |