| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-04 | |||
| 13:53:49 | mriedem | "add the column to the whitelist so it will be ignored" | |
| 13:54:06 | alex_xu | dansmith: what the whitelist mean? | |
| 13:54:26 | dansmith | alex_xu: look in those scheduled_at patches.. there's a whitelist for the test that validates that the model and schema match | |
| 13:55:03 | dansmith | artom: looks like the ci job on top never really completes because sean-k-mooney is working on it.. got a link to a recent run? | |
| 13:56:07 | artom | dansmith, so I left a comment on PS7 of https://review.opendev.org/#/c/679656/ linking to various signs of success | |
| 13:56:12 | sean-k-mooney | dansmith: well i just started updateing them again today since there were no running jobs | |
| 13:56:15 | artom | lemme see if I can find something more recent | |
| 13:56:31 | dansmith | artom: yeah that was before the hugepage work right? | |
| 13:56:43 | artom | dansmith, I believe that was after | |
| 13:56:57 | sean-k-mooney | this was after https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz | |
| 13:57:01 | artom | dansmith, 13:12 my time, I was talking to danpb that morning about hugepages | |
| 13:58:05 | sean-k-mooney | this is from the run where i forced the vcpu_pin_set and cpu_shared_set to be reversed on each host | |
| 13:58:07 | sean-k-mooney | https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/compute/logs/etc/nova/nova_conf.txt.gz | |
| 13:59:12 | luyao | dansmith, alex_xu : I found the whitelist. Thanks! | |
| 13:59:31 | alex_xu | luyao: really, I'm not...still looking | |
| 14:00:15 | alex_xu | ah, I found it | |
| 14:00:23 | alex_xu | luyao: you win | |
| 14:00:23 | luyao | alex_xu: it's in nova/tests/unit/db/test_migrations.py, def filter_metadata_diff | |
| 14:00:34 | alex_xu | dansmith: thanks again | |
| 14:02:18 | yonglihe | alex_xu: Good evening. | |
| 14:02:29 | alex_xu | yonglihe: good evening too :) | |
| 14:09:26 | dansmith | artom: sean-k-mooney: What're ya'll doing to test the rollback case IRL? | |
| 14:10:00 | artom | dansmith, IRL? nothing. Trying to understand why the func test wasn't picking it up | |
| 14:10:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove Rocky compute compat checks for live migration with port bindings https://review.opendev.org/680005 | |
| 14:10:23 | dansmith | artom: okay, I would expect you'd set up something to fail the migration to make sure the rollback actually happens properly | |
| 14:10:27 | sean-k-mooney | e.g. a failure to migrate, i have inetvertenly tested it by forgettting to exchanged ssh keys and haveing the migrate fail and rollback | |
| 14:10:39 | sean-k-mooney | that is the only testing of it i did | |
| 14:10:43 | dansmith | sean-k-mooney: is it getting far enough that it actually calls the new method? | |
| 14:10:54 | sean-k-mooney | yes | |
| 14:10:58 | dansmith | okay | |
| 14:11:03 | sean-k-mooney | i have not done that on the latest version | |
| 14:11:08 | sean-k-mooney | want me to do that | |
| 14:11:23 | sean-k-mooney | i just need to drop the ssh key form autherised keys | |
| 14:11:30 | dansmith | sean-k-mooney: can you also do something else for us? pin the compute version on one machine to 5.2 and migrate both directions to make sure it still works | |
| 14:11:40 | dansmith | because we really *really* don't want to break migrations for upgrades | |
| 14:11:47 | artom | dansmith, I suppose I can setup stein/master and try it | |
| 14:11:53 | sean-k-mooney | sure. i can do that in a ci test too | |
| 14:12:06 | artom | dansmith, oh, just pinning. yeah :) | |
| 14:12:06 | dansmith | if we find out late that we did and need to tweak rpc stuff to make it work, life is gonna suck | |
| 14:12:08 | sean-k-mooney | but ill test both | |
| 14:12:15 | dansmith | sean-k-mooney: okay thanks | |
| 14:12:33 | dansmith | I see the logs show it actually doing the new stuff so that's cool | |
| 14:12:33 | sean-k-mooney | dansmith: i can also give you acess once i have it set up if you want to poke around | |
| 14:12:43 | dansmith | sean-k-mooney: that's okay, I' | |
| 14:12:44 | artom | (Wouldn't func be sufficient - like, I get wanting to see real libvirt with the updated XML, and thank god because it caught bugs) | |
| 14:12:50 | dansmith | sean-k-mooney: I'll trust some pastebin'd logs | |
| 14:13:01 | artom | (But rollback is all compute) | |
| 14:13:02 | dansmith | artom: func is great, but it's not the real thing | |
| 14:13:26 | dansmith | artom: trust me, you don't want to be on a sev1 where you realize you're going to need to add an RPC .z version to fix this | |
| 14:13:34 | dansmith | we did that in icehouse for a customer and it was not cool | |
| 14:13:48 | artom | Fully agree there | |
| 14:14:57 | artom | sean-k-mooney, how feasible would it be to modify the grenade job to run with a NUMA flavor? | |
| 14:15:04 | artom | On FN hardwa | |
| 14:15:06 | artom | *hardware | |
| 14:15:33 | sean-k-mooney | so i am less familar with legacy jobs but it could be done | |
| 14:15:54 | sean-k-mooney | the nodeset is not hard but forcing it to use nested virt is the tricky bit | |
| 14:16:05 | sean-k-mooney | devstack gate is non tivial | |
| 14:16:14 | sean-k-mooney | adding grendate on top is harder | |
| 14:16:22 | sean-k-mooney | if we need to do that we could however | |
| 14:16:43 | artom | sean-k-mooney, well, dansmith said he's satisfied with pastebin, so maybe more long term | |
| 14:16:44 | sean-k-mooney | i would like to port greadne to a non legacy job at some point | |
| 14:17:23 | sean-k-mooney | well well i have not updated https://review.opendev.org/#/c/679805/ to pick up the other changes i did this morning | |
| 14:17:41 | sean-k-mooney | i can updated it and add the different compute versions in that | |
| 14:17:55 | sean-k-mooney | ill try and test it locally first however | |
| 14:19:44 | mriedem | speaking of converting a job to zuulv3, https://review.opendev.org/#/c/674345/ converts the nova-lvm job to zuulv3 and has a +2 | |
| 14:19:46 | sean-k-mooney | dansmith: do you know how to pin the rpc version of the top of your head. ill look it up in teh config ref if not | |
| 14:20:00 | mriedem | sean-k-mooney: [upgrade_levels]/compute=stein | |
| 14:20:05 | dansmith | yeah that | |
| 14:20:10 | sean-k-mooney | thanks | |
| 14:20:24 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#upgrade_levels.compute | |
| 14:21:24 | mriedem | once that nova-lvm change is merged the only legacy jobs we have are the live migration and grenade multinode ones, which i've probably put off b/c of the external script they run, but nova-next was converted and runs an external script so i can probably do similar | |
| 14:21:24 | sean-k-mooney | does it matter if i do this on the compute or the contoler? e.g. we want the conductor ect to not be pinned right | |
| 14:21:31 | sean-k-mooney | just the source node? | |
| 14:21:38 | mriedem | sean-k-mooney: if you pin in nova-cpu.conf i don't think it matters | |
| 14:21:44 | mriedem | conductor is using a different config | |
| 14:22:14 | sean-k-mooney | ya just want to make sure i dont mess up my local.conf and have to redeploy | |
| 14:22:27 | sean-k-mooney | ill pin it on the dedicated compute node | |
| 14:22:39 | sean-k-mooney | nova-cpu.conf would also work | |
| 14:23:08 | sean-k-mooney | yes you did i was agreeing with you | |
| 14:24:08 | mriedem | i can tinker later this afternoon with converting that nova-grenade-multinode job to zuulv3 | |
| 14:24:13 | mriedem | those have been on my todo list for awhile | |
| 14:25:42 | sean-k-mooney | i can also try if you run into issue but i have never actully run greade and i dont know if we have the ansibel playbooks to run it so im not sure how easy that will be | |
| 14:26:17 | sean-k-mooney | is greade hard to run or is basically the same as devstack | |
| 14:32:03 | mriedem | it's just a set of bash scripts that orchestrates setting up n-1 devstack, creating some resources, shutting down and then running N version devstack and making sure those resources are still around | |
| 14:32:14 | mriedem | i've never run it locally manually, only in CI | |
| 14:32:33 | gibi | mriedem: looking at the MigrationTask. When the dest is selected by the scheduler I can check right away if the dest has a new enough compute service version. But if not, then what? simulate a re-schedule locally in the conductor? https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/conductor/tasks/migrate.py#L248 | |
| 14:33:10 | mriedem | gibi: you'd have to iterate alternates | |
| 14:33:13 | sean-k-mooney | mriedem: looking at the zuul config they dont have a non legacy way to run it | |
| 14:33:19 | sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/.zuul.yaml | |
| 14:33:21 | mriedem | it's times like this that i've thought about having a compute service version filter in the scheduler | |
| 14:33:32 | mriedem | so we can say internally "only give me computes with service version >= x for this feature" | |
| 14:34:12 | sean-k-mooney | mriedem: well if that feature was a compute capablity tratis we could use placemtn for that | |
| 14:34:14 | gibi | mriedem: Yeah that was what I mean by simualting a re-schedule locally , as I have to take care of num_retries and the allocations too | |
| 14:34:16 | mriedem | sean-k-mooney: https://review.opendev.org/#/c/548936/ | |
| 14:34:20 | sean-k-mooney | like your multi attach stuff | |
| 14:34:39 | openstackgerrit | Merged openstack/nova master: Make _get_cpu_feature_traits() always return a dict https://review.opendev.org/679568 | |
| 14:34:52 | sean-k-mooney | mriedem: ya i saw that mentioned on the infra irc yesterday | |
| 14:34:52 | openstackgerrit | Merged openstack/nova master: libvirt: Make scheduler filters customizable https://review.opendev.org/679745 | |
| 14:35:00 | sean-k-mooney | i was not sure how far along it was | |
| 14:35:00 | mriedem | gibi: num_retries, you mean max_attempts? | |