| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-04 | |||
| 13:40:56 | dansmith | the RPC versioning scheme that we have is why, in many cases, you actually can run N-2 talking to N when things haven't shifted much, and what would save our bacon if we had to start supporting such a thing | |
| 13:41:04 | dansmith | upstream doesn't, but distros have in the past for sure | |
| 13:41:05 | dansmith | however, | |
| 13:41:29 | dansmith | it's easy to backport something that doesn't technically affect the rpc interface directly, but definitely changes behavior or semantics | |
| 13:41:41 | dansmith | like something that causes a field to be filled out that wouldn't have been if the client was really as old as it is, etc | |
| 13:42:04 | dansmith | like some distro that wanted to backport an api feature that now sends something newer, but doesn't technically need an rpc change to make it work | |
| 13:42:25 | artom | So the reason the first rule doesn't invalidate the second rule is that even if we only allow a mix of N and N-1, there's nothing stopping an older RPC pin from being set? | |
| 13:42:51 | dansmith | yes, but let me state it slightly differently: | |
| 13:43:19 | dansmith | the first rule is our support policy and the second is the actual mechanical way we make sure our services can sanely talk to each other across versions | |
| 13:43:37 | artom | Ah, I see | |
| 13:43:56 | dansmith | so theoretically the mechanical bit should be able to support a wider envelope (rule 2) than we officially say we support (rule 1) | |
| 13:44:06 | artom | IOW, we don't have mechanics in place to enforce our support policy, short of bumping major on every release | |
| 13:44:14 | dansmith | because the N->N+2 thing has lots of other implications as well, like database migrations, things that need to check service versions to enable api features, etc, etc | |
| 13:44:23 | dansmith | that's a true statement | |
| 13:45:11 | artom | Aha, so the support policy is not just for RPC, it's for other things as well (that list you just said) | |
| 13:45:33 | dansmith | yup | |
| 13:45:53 | dansmith | because the other things are buried deep and don't necessarily even have a direct impact on rpc, but can definitely have an indirect one | |
| 13:46:23 | dansmith | alright, so I've passed the torch of knowledge, now you can be the enforcer right? | |
| 13:46:36 | artom | dansmith, not sure about that, but I'll write it up in https://docs.openstack.org/nova/stein/reference/index.html | |
| 13:46:51 | mriedem | it's already written up in dan's blog... | |
| 13:47:04 | mriedem | if you wanted to condense that, sure | |
| 13:47:06 | dansmith | but writing it up in our docs would be good | |
| 13:47:24 | mriedem | what i was thinking about was https://review.opendev.org/#/c/634606/73/nova/compute/manager.py@6574 specifically is only needed for train, | |
| 13:47:30 | mriedem | since we don't support Stein -> U | |
| 13:47:34 | dansmith | if my blog is part of the official nova docs, I'm going to throw some ads up there :) | |
| 13:47:47 | artom | dansmith, for AWS :D | |
| 13:48:08 | dansmith | mriedem: that's only the case because of the rule 1 above | |
| 13:48:29 | mriedem | sure, | |
| 13:48:33 | dansmith | mriedem: they can still set the pin lower than train in U and would fail to make that call.. it'd be an unsupported situation, but it's probably good if we just leave it in place | |
| 13:48:40 | mriedem | and law of nova #35 says we'll forget to remove the compat code for 3 releases anyway | |
| 13:48:49 | dansmith | heh | |
| 13:49:08 | dansmith | I can tell you what rule #1 is | |
| 13:49:27 | mriedem | there is a rule somewhere that you must have at least 3 different types of BlockDevice(Mapping) objects | |
| 13:49:37 | dansmith | #17 iirc | |
| 13:49:59 | artom | dansmith, you left us hanging on #1 there | |
| 13:50:08 | dansmith | artom: don't talk about nova, obvi | |
| 13:50:32 | artom | OK, I lied. | |
| 13:50:58 | artom | dansmith, mriedem, just so I'm clear, are we waiting on anything from me and/or CI for the NUMA LM series? | |
| 13:50:58 | alex_xu | dansmith: let me know I understand this rule correct or not https://review.opendev.org/#/c/678447/7/nova/db/sqlalchemy/migrate_repo/versions/401_add_resources.py@32 :) | |
| 13:51:32 | dansmith | artom: nope, I had it up to be looking at when I saw your ping | |
| 13:51:34 | mriedem | i've been online for ~45 minutes and already have a full tab queue, so no | |
| 13:51:42 | luyao | dansmith: Hi, I have a question about https://review.opendev.org/#/c/678447/8 . Do you mean the script can run out of the nova code, the script will just update DB, then db column is deleted while the updates on nova code have not taken effect.I don't understand how I should do. | |
| 13:52:02 | alex_xu | luyao: i just asked :) | |
| 13:52:28 | artom | mriedem, yep, fully away of your overcommitment, as just making sure I understood where we stood :) | |
| 13:52:31 | artom | *was | |
| 13:52:36 | dansmith | alex_xu: yes, scheduled_at is the only field we've removed in a long time since we started doing the model/schema sync, so follow that pattern | |
| 13:53:11 | alex_xu | dansmith: ah, got it, leave comment on the model obj, and rename the exisiting field to resources, then remove it next release | |
| 13:53:21 | alex_xu | dansmith: thanks! | |
| 13:53:34 | luyao | dansmith: I got this fail on test_models_sync , AssertionError: Models and migration scripts aren't in sync: | |
| 13:53:34 | dansmith | alex_xu: rename the field on the model, but you have to add the column to the whitelist so it will be ignored, but yeah. | |
| 13:53:49 | mriedem | "add the column to the whitelist so it will be ignored" | |
| 13:54:06 | alex_xu | dansmith: what the whitelist mean? | |
| 13:54:26 | dansmith | alex_xu: look in those scheduled_at patches.. there's a whitelist for the test that validates that the model and schema match | |
| 13:55:03 | dansmith | artom: looks like the ci job on top never really completes because sean-k-mooney is working on it.. got a link to a recent run? | |
| 13:56:07 | artom | dansmith, so I left a comment on PS7 of https://review.opendev.org/#/c/679656/ linking to various signs of success | |
| 13:56:12 | sean-k-mooney | dansmith: well i just started updateing them again today since there were no running jobs | |
| 13:56:15 | artom | lemme see if I can find something more recent | |
| 13:56:31 | dansmith | artom: yeah that was before the hugepage work right? | |
| 13:56:43 | artom | dansmith, I believe that was after | |
| 13:56:57 | sean-k-mooney | this was after https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz | |
| 13:57:01 | artom | dansmith, 13:12 my time, I was talking to danpb that morning about hugepages | |
| 13:58:05 | sean-k-mooney | this is from the run where i forced the vcpu_pin_set and cpu_shared_set to be reversed on each host | |
| 13:58:07 | sean-k-mooney | https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/compute/logs/etc/nova/nova_conf.txt.gz | |
| 13:59:12 | luyao | dansmith, alex_xu : I found the whitelist. Thanks! | |
| 13:59:31 | alex_xu | luyao: really, I'm not...still looking | |
| 14:00:15 | alex_xu | ah, I found it | |
| 14:00:23 | alex_xu | luyao: you win | |
| 14:00:23 | luyao | alex_xu: it's in nova/tests/unit/db/test_migrations.py, def filter_metadata_diff | |
| 14:00:34 | alex_xu | dansmith: thanks again | |
| 14:02:18 | yonglihe | alex_xu: Good evening. | |
| 14:02:29 | alex_xu | yonglihe: good evening too :) | |
| 14:09:26 | dansmith | artom: sean-k-mooney: What're ya'll doing to test the rollback case IRL? | |
| 14:10:00 | artom | dansmith, IRL? nothing. Trying to understand why the func test wasn't picking it up | |
| 14:10:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove Rocky compute compat checks for live migration with port bindings https://review.opendev.org/680005 | |
| 14:10:23 | dansmith | artom: okay, I would expect you'd set up something to fail the migration to make sure the rollback actually happens properly | |
| 14:10:27 | sean-k-mooney | e.g. a failure to migrate, i have inetvertenly tested it by forgettting to exchanged ssh keys and haveing the migrate fail and rollback | |
| 14:10:39 | sean-k-mooney | that is the only testing of it i did | |
| 14:10:43 | dansmith | sean-k-mooney: is it getting far enough that it actually calls the new method? | |
| 14:10:54 | sean-k-mooney | yes | |
| 14:10:58 | dansmith | okay | |
| 14:11:03 | sean-k-mooney | i have not done that on the latest version | |
| 14:11:08 | sean-k-mooney | want me to do that | |
| 14:11:23 | sean-k-mooney | i just need to drop the ssh key form autherised keys | |
| 14:11:30 | dansmith | sean-k-mooney: can you also do something else for us? pin the compute version on one machine to 5.2 and migrate both directions to make sure it still works | |
| 14:11:40 | dansmith | because we really *really* don't want to break migrations for upgrades | |
| 14:11:47 | artom | dansmith, I suppose I can setup stein/master and try it | |
| 14:11:53 | sean-k-mooney | sure. i can do that in a ci test too | |
| 14:12:06 | artom | dansmith, oh, just pinning. yeah :) | |
| 14:12:06 | dansmith | if we find out late that we did and need to tweak rpc stuff to make it work, life is gonna suck | |
| 14:12:08 | sean-k-mooney | but ill test both | |
| 14:12:15 | dansmith | sean-k-mooney: okay thanks | |
| 14:12:33 | dansmith | I see the logs show it actually doing the new stuff so that's cool | |
| 14:12:33 | sean-k-mooney | dansmith: i can also give you acess once i have it set up if you want to poke around | |
| 14:12:43 | dansmith | sean-k-mooney: that's okay, I' | |
| 14:12:44 | artom | (Wouldn't func be sufficient - like, I get wanting to see real libvirt with the updated XML, and thank god because it caught bugs) | |
| 14:12:50 | dansmith | sean-k-mooney: I'll trust some pastebin'd logs | |
| 14:13:01 | artom | (But rollback is all compute) | |
| 14:13:02 | dansmith | artom: func is great, but it's not the real thing | |
| 14:13:26 | dansmith | artom: trust me, you don't want to be on a sev1 where you realize you're going to need to add an RPC .z version to fix this | |
| 14:13:34 | dansmith | we did that in icehouse for a customer and it was not cool | |