Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-04
13:24:57 mriedem i have to take my kid to the bus stop, will look into this later
13:25:09 gmann ohk, did not see the previous comment
13:25:41 brinzhang gmann: that will be change the rpc version, in PS13 do that
13:26:05 gibi bauzas: It might take more time to update the bandwidth series than I first aniticipated. I will ping you when the new version is up
13:26:13 gmann i did not see that, checking..
13:28:10 bauzas gibi: no worries, I'm still working on updating the placement audit command
13:35:26 dansmith artom: to what are you referring?
13:36:11 artom dansmith, all those checks I have about either making sure we got a param that appears in RPC 5.3, or that we can send RPC 5.3, those can only be removed in the next major version bump, right?
13:36:29 artom dansmith, if so, mind explaining why, exactly? It's still not clear in my head why they can't just go away in U
13:37:05 dansmith well, it's because there are two sets of "rules" at play here:
13:37:25 dansmith The first is that we don't allow RPC upgrades from N-2 to N in terms of releases
13:38:00 dansmith we do that not just because of strict rpc compatibility, but because of stuff buried deeper that expects fields to be filled out two releases from now, data migrations to have run, etc
13:38:13 artom dansmith, this means that we don't support Train anything talking to Rocky anything, right? Only Stein
13:39:18 dansmith the second is that the RPC versioning rules say that within a major version, you must be backwards compatible, able to accept anything back to 5.0 and able to send anything back to 5.0 if asked (by the version pin). If we always bumped versions on every release, then you could always remove stuff two releases from when you added it (since each release would have to bridge between two majors, you could never remove it in the next one)
13:39:59 dansmith artom: that's what it means, but since the rpc version is all we have, it's really a soft statement of support that we can't validate or quantify,
13:40:04 dansmith especially since people backport stuff
13:40:31 artom People backport RPC stuff?
13:40:56 dansmith the RPC versioning scheme that we have is why, in many cases, you actually can run N-2 talking to N when things haven't shifted much, and what would save our bacon if we had to start supporting such a thing
13:41:04 dansmith upstream doesn't, but distros have in the past for sure
13:41:05 dansmith however,
13:41:29 dansmith it's easy to backport something that doesn't technically affect the rpc interface directly, but definitely changes behavior or semantics
13:41:41 dansmith like something that causes a field to be filled out that wouldn't have been if the client was really as old as it is, etc
13:42:04 dansmith like some distro that wanted to backport an api feature that now sends something newer, but doesn't technically need an rpc change to make it work
13:42:25 artom So the reason the first rule doesn't invalidate the second rule is that even if we only allow a mix of N and N-1, there's nothing stopping an older RPC pin from being set?
13:42:51 dansmith yes, but let me state it slightly differently:
13:43:19 dansmith the first rule is our support policy and the second is the actual mechanical way we make sure our services can sanely talk to each other across versions
13:43:37 artom Ah, I see
13:43:56 dansmith so theoretically the mechanical bit should be able to support a wider envelope (rule 2) than we officially say we support (rule 1)
13:44:06 artom IOW, we don't have mechanics in place to enforce our support policy, short of bumping major on every release
13:44:14 dansmith because the N->N+2 thing has lots of other implications as well, like database migrations, things that need to check service versions to enable api features, etc, etc
13:44:23 dansmith that's a true statement
13:45:11 artom Aha, so the support policy is not just for RPC, it's for other things as well (that list you just said)
13:45:33 dansmith yup
13:45:53 dansmith because the other things are buried deep and don't necessarily even have a direct impact on rpc, but can definitely have an indirect one
13:46:23 dansmith alright, so I've passed the torch of knowledge, now you can be the enforcer right?
13:46:36 artom dansmith, not sure about that, but I'll write it up in https://docs.openstack.org/nova/stein/reference/index.html
13:46:51 mriedem it's already written up in dan's blog...
13:47:04 mriedem if you wanted to condense that, sure
13:47:06 dansmith but writing it up in our docs would be good
13:47:24 mriedem what i was thinking about was https://review.opendev.org/#/c/634606/73/nova/compute/manager.py@6574 specifically is only needed for train,
13:47:30 mriedem since we don't support Stein -> U
13:47:34 dansmith if my blog is part of the official nova docs, I'm going to throw some ads up there :)
13:47:47 artom dansmith, for AWS :D
13:48:08 dansmith mriedem: that's only the case because of the rule 1 above
13:48:29 mriedem sure,
13:48:33 dansmith mriedem: they can still set the pin lower than train in U and would fail to make that call.. it'd be an unsupported situation, but it's probably good if we just leave it in place
13:48:40 mriedem and law of nova #35 says we'll forget to remove the compat code for 3 releases anyway
13:48:49 dansmith heh
13:49:08 dansmith I can tell you what rule #1 is
13:49:27 mriedem there is a rule somewhere that you must have at least 3 different types of BlockDevice(Mapping) objects
13:49:37 dansmith #17 iirc
13:49:59 artom dansmith, you left us hanging on #1 there
13:50:08 dansmith artom: don't talk about nova, obvi
13:50:32 artom OK, I lied.
13:50:58 alex_xu dansmith: let me know I understand this rule correct or not https://review.opendev.org/#/c/678447/7/nova/db/sqlalchemy/migrate_repo/versions/401_add_resources.py@32 :)
13:50:58 artom dansmith, mriedem, just so I'm clear, are we waiting on anything from me and/or CI for the NUMA LM series?
13:51:32 dansmith artom: nope, I had it up to be looking at when I saw your ping
13:51:34 mriedem i've been online for ~45 minutes and already have a full tab queue, so no
13:51:42 luyao dansmith: Hi, I have a question about https://review.opendev.org/#/c/678447/8 . Do you mean the script can run out of the nova code, the script will just update DB, then db column is deleted while the updates on nova code have not taken effect.I don't understand how I should do.
13:52:02 alex_xu luyao: i just asked :)
13:52:28 artom mriedem, yep, fully away of your overcommitment, as just making sure I understood where we stood :)
13:52:31 artom *was
13:52:36 dansmith alex_xu: yes, scheduled_at is the only field we've removed in a long time since we started doing the model/schema sync, so follow that pattern
13:53:11 alex_xu dansmith: ah, got it, leave comment on the model obj, and rename the exisiting field to resources, then remove it next release
13:53:21 alex_xu dansmith: thanks!
13:53:34 dansmith alex_xu: rename the field on the model, but you have to add the column to the whitelist so it will be ignored, but yeah.
13:53:34 luyao dansmith: I got this fail on test_models_sync , AssertionError: Models and migration scripts aren't in sync:
13:53:49 mriedem "add the column to the whitelist so it will be ignored"
13:54:06 alex_xu dansmith: what the whitelist mean?
13:54:26 dansmith alex_xu: look in those scheduled_at patches.. there's a whitelist for the test that validates that the model and schema match
13:55:03 dansmith artom: looks like the ci job on top never really completes because sean-k-mooney is working on it.. got a link to a recent run?
13:56:07 artom dansmith, so I left a comment on PS7 of https://review.opendev.org/#/c/679656/ linking to various signs of success
13:56:12 sean-k-mooney dansmith: well i just started updateing them again today since there were no running jobs
13:56:15 artom lemme see if I can find something more recent
13:56:31 dansmith artom: yeah that was before the hugepage work right?
13:56:43 artom dansmith, I believe that was after
13:56:57 sean-k-mooney this was after https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz
13:57:01 artom dansmith, 13:12 my time, I was talking to danpb that morning about hugepages
13:58:05 sean-k-mooney this is from the run where i forced the vcpu_pin_set and cpu_shared_set to be reversed on each host
13:58:07 sean-k-mooney https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/compute/logs/etc/nova/nova_conf.txt.gz
13:59:12 luyao dansmith, alex_xu : I found the whitelist. Thanks!
13:59:31 alex_xu luyao: really, I'm not...still looking
14:00:15 alex_xu ah, I found it
14:00:23 luyao alex_xu: it's in nova/tests/unit/db/test_migrations.py, def filter_metadata_diff
14:00:23 alex_xu luyao: you win
14:00:34 alex_xu dansmith: thanks again
14:02:18 yonglihe alex_xu: Good evening.
14:02:29 alex_xu yonglihe: good evening too :)
14:09:26 dansmith artom: sean-k-mooney: What're ya'll doing to test the rollback case IRL?
14:10:00 artom dansmith, IRL? nothing. Trying to understand why the func test wasn't picking it up
14:10:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove Rocky compute compat checks for live migration with port bindings https://review.opendev.org/680005
14:10:23 dansmith artom: okay, I would expect you'd set up something to fail the migration to make sure the rollback actually happens properly
14:10:27 sean-k-mooney e.g. a failure to migrate, i have inetvertenly tested it by forgettting to exchanged ssh keys and haveing the migrate fail and rollback
14:10:39 sean-k-mooney that is the only testing of it i did
14:10:43 dansmith sean-k-mooney: is it getting far enough that it actually calls the new method?
14:10:54 sean-k-mooney yes
14:10:58 dansmith okay
14:11:03 sean-k-mooney i have not done that on the latest version
14:11:08 sean-k-mooney want me to do that
14:11:23 sean-k-mooney i just need to drop the ssh key form autherised keys

Earlier   Later